2024/11/19-15:14:31.433834 139852449619904 RocksDB version: 8.4.2 2024/11/19-15:14:31.433879 139852449619904 Git sha 1436c4478ab05164c1d0929db1403a3077b1ba12 2024/11/19-15:14:31.433886 139852449619904 Compile date 2024-11-18 13:58:37 2024/11/19-15:14:31.433895 139852449619904 DB SUMMARY 2024/11/19-15:14:31.433901 139852449619904 Host name (Env): iZwz9deglxdazgiryp6ltbZ 2024/11/19-15:14:31.433906 139852449619904 DB Session ID: MU9ZNZIP7NRMU2YPYQ8B 2024/11/19-15:14:31.434055 139852449619904 CURRENT file: CURRENT 2024/11/19-15:14:31.434062 139852449619904 IDENTITY file: IDENTITY 2024/11/19-15:14:31.434069 139852449619904 MANIFEST file: MANIFEST-004299 size: 28919 Bytes 2024/11/19-15:14:31.434074 139852449619904 SST files in /storage/mytopling/.rocksdb dir, Total Num: 230, files: 003692.sst 003697.sst 003703.sst 003707.sst 003708.sst 003709.sst 003712.sst 003713.sst 003714.sst 2024/11/19-15:14:31.434081 139852449619904 Write Ahead Log file in /storage/mytopling/.rocksdb: 004316.log size: 0 ; 2024/11/19-15:14:31.434087 139852449619904 Options.error_if_exists: 0 2024/11/19-15:14:31.434092 139852449619904 Options.create_if_missing: 1 2024/11/19-15:14:31.434097 139852449619904 Options.paranoid_checks: 1 2024/11/19-15:14:31.434102 139852449619904 Options.flush_verify_memtable_count: 1 2024/11/19-15:14:31.434106 139852449619904 Options.track_and_verify_wals_in_manifest: 1 2024/11/19-15:14:31.434111 139852449619904 Options.verify_sst_unique_id_in_manifest: 1 2024/11/19-15:14:31.434116 139852449619904 Options.env: 0x16e21280 2024/11/19-15:14:31.434121 139852449619904 Options.fs: PosixFileSystem 2024/11/19-15:14:31.434126 139852449619904 Options.info_log: 0x170c8150 2024/11/19-15:14:31.434130 139852449619904 Options.max_file_opening_threads: 16 2024/11/19-15:14:31.434137 139852449619904 Options.statistics: 0x16e302e0 2024/11/19-15:14:31.434144 139852449619904 Options.statistics stats level: 3 2024/11/19-15:14:31.434150 139852449619904 Options.use_fsync: 0 2024/11/19-15:14:31.434154 139852449619904 Options.max_log_file_size: 268435456 2024/11/19-15:14:31.434162 139852449619904 Options.max_manifest_file_size: 1073741824 2024/11/19-15:14:31.434168 139852449619904 Options.log_file_time_to_roll: 0 2024/11/19-15:14:31.434179 139852449619904 Options.keep_log_file_num: 100 2024/11/19-15:14:31.434185 139852449619904 Options.recycle_log_file_num: 0 2024/11/19-15:14:31.434192 139852449619904 Options.allow_fallocate: 1 2024/11/19-15:14:31.434196 139852449619904 Options.allow_mmap_reads: 0 2024/11/19-15:14:31.434210 139852449619904 Options.allow_mmap_writes: 0 2024/11/19-15:14:31.434216 139852449619904 Options.use_direct_reads: 0 2024/11/19-15:14:31.434223 139852449619904 Options.use_direct_io_for_flush_and_compaction: 0 2024/11/19-15:14:31.434229 139852449619904 Options.create_missing_column_families: 1 2024/11/19-15:14:31.434236 139852449619904 Options.db_log_dir: /storage/mytopling/..//infolog/.rocksdb 2024/11/19-15:14:31.434240 139852449619904 Options.wal_dir: 2024/11/19-15:14:31.434247 139852449619904 Options.table_cache_numshardbits: 6 2024/11/19-15:14:31.434253 139852449619904 Options.WAL_ttl_seconds: 0 2024/11/19-15:14:31.434260 139852449619904 Options.WAL_size_limit_MB: 0 2024/11/19-15:14:31.434266 139852449619904 Options.max_write_batch_group_size_bytes: 1048576 2024/11/19-15:14:31.434273 139852449619904 Options.manifest_preallocation_size: 4194304 2024/11/19-15:14:31.434277 139852449619904 Options.is_fd_close_on_exec: 1 2024/11/19-15:14:31.434284 139852449619904 Options.advise_random_on_open: 1 2024/11/19-15:14:31.434290 139852449619904 Options.db_write_buffer_size: 0 2024/11/19-15:14:31.434296 139852449619904 Options.write_buffer_manager: 0x170e7c70 2024/11/19-15:14:31.434303 139852449619904 Options.access_hint_on_compaction_start: 1 2024/11/19-15:14:31.434310 139852449619904 Options.random_access_max_buffer_size: 1048576 2024/11/19-15:14:31.434314 139852449619904 Options.use_adaptive_mutex: 0 2024/11/19-15:14:31.434320 139852449619904 Options.rate_limiter: (nil) 2024/11/19-15:14:31.434328 139852449619904 Options.sst_file_manager.rate_bytes_per_sec: 0 2024/11/19-15:14:31.434335 139852449619904 Options.wal_recovery_mode: 2 2024/11/19-15:14:31.434341 139852449619904 Options.enable_thread_tracking: 1 2024/11/19-15:14:31.434348 139852449619904 Options.enable_pipelined_write: 0 2024/11/19-15:14:31.434352 139852449619904 Options.unordered_write: 0 2024/11/19-15:14:31.434358 139852449619904 Options.allow_concurrent_memtable_write: 1 2024/11/19-15:14:31.434364 139852449619904 Options.enable_write_thread_adaptive_yield: 0 2024/11/19-15:14:31.434371 139852449619904 Options.write_thread_max_yield_usec: 100 2024/11/19-15:14:31.434381 139852449619904 Options.write_thread_slow_yield_usec: 3 2024/11/19-15:14:31.434387 139852449619904 Options.row_cache: None 2024/11/19-15:14:31.434392 139852449619904 Options.wal_filter: None 2024/11/19-15:14:31.434398 139852449619904 Options.avoid_flush_during_recovery: 0 2024/11/19-15:14:31.434404 139852449619904 Options.allow_ingest_behind: 0 2024/11/19-15:14:31.434411 139852449619904 Options.two_write_queues: 1 2024/11/19-15:14:31.434419 139852449619904 Options.manual_wal_flush: 1 2024/11/19-15:14:31.434426 139852449619904 Options.wal_compression: 0 2024/11/19-15:14:31.434430 139852449619904 Options.atomic_flush: 0 2024/11/19-15:14:31.434436 139852449619904 Options.avoid_unnecessary_blocking_io: 1 2024/11/19-15:14:31.434443 139852449619904 Options.persist_stats_to_disk: 0 2024/11/19-15:14:31.434449 139852449619904 Options.write_dbid_to_manifest: 1 2024/11/19-15:14:31.434455 139852449619904 Options.log_readahead_size: 0 2024/11/19-15:14:31.434462 139852449619904 Options.file_checksum_gen_factory: Unknown 2024/11/19-15:14:31.434466 139852449619904 Options.best_efforts_recovery: 0 2024/11/19-15:14:31.434473 139852449619904 Options.max_bgerror_resume_count: 2147483647 2024/11/19-15:14:31.434479 139852449619904 Options.bgerror_resume_retry_interval: 1000000 2024/11/19-15:14:31.434486 139852449619904 Options.allow_data_in_errors: 0 2024/11/19-15:14:31.434492 139852449619904 Options.db_host_id: __hostname__ 2024/11/19-15:14:31.434499 139852449619904 Options.enforce_single_del_contracts: false 2024/11/19-15:14:31.434504 139852449619904 Options.max_background_jobs: 2 2024/11/19-15:14:31.434510 139852449619904 Options.max_background_compactions: 7 2024/11/19-15:14:31.434517 139852449619904 Options.max_subcompactions: 1 2024/11/19-15:14:31.434523 139852449619904 Options.max_level1_subcompactions: 3 2024/11/19-15:14:31.434530 139852449619904 Options.avoid_flush_during_shutdown: 0 2024/11/19-15:14:31.434536 139852449619904 Options.writable_file_max_buffer_size: 1048576 2024/11/19-15:14:31.434542 139852449619904 Options.delayed_write_rate : 16777216 2024/11/19-15:14:31.434547 139852449619904 Options.max_total_wal_size: 0 2024/11/19-15:14:31.434553 139852449619904 Options.delete_obsolete_files_period_micros: 21600000000 2024/11/19-15:14:31.434562 139852449619904 Options.stats_dump_period_sec: 0 2024/11/19-15:14:31.434568 139852449619904 Options.stats_persist_period_sec: 0 2024/11/19-15:14:31.434574 139852449619904 Options.stats_history_buffer_size: 1048576 2024/11/19-15:14:31.434581 139852449619904 Options.max_open_files: -1 2024/11/19-15:14:31.434585 139852449619904 Options.bytes_per_sync: 0 2024/11/19-15:14:31.434592 139852449619904 Options.wal_bytes_per_sync: 0 2024/11/19-15:14:31.434598 139852449619904 Options.strict_bytes_per_sync: 0 2024/11/19-15:14:31.434604 139852449619904 Options.compaction_readahead_size: 0 2024/11/19-15:14:31.434611 139852449619904 Options.max_background_flushes: 2 2024/11/19-15:14:31.434617 139852449619904 Compression algorithms supported: 2024/11/19-15:14:31.434622 139852449619904 kZSTDNotFinalCompression supported: 1 2024/11/19-15:14:31.434632 139852449619904 kZSTD supported: 1 2024/11/19-15:14:31.434639 139852449619904 kXpressCompression supported: 0 2024/11/19-15:14:31.434648 139852449619904 kLZ4HCCompression supported: 1 2024/11/19-15:14:31.434655 139852449619904 kZlibCompression supported: 1 2024/11/19-15:14:31.434670 139852449619904 kSnappyCompression supported: 1 2024/11/19-15:14:31.434675 139852449619904 kLZ4Compression supported: 1 2024/11/19-15:14:31.434681 139852449619904 kBZip2Compression supported: 1 2024/11/19-15:14:31.434701 139852449619904 Fast CRC32 supported: Supported on x86 2024/11/19-15:14:31.434708 139852449619904 DMutex implementation: pthread_mutex_t 2024/11/19-15:14:31.434786 139852449619904 [version_set.cc:6211] Recovering from manifest file: /storage/mytopling/.rocksdb/MANIFEST-004299 2024/11/19-15:14:31.434972 139852449619904 [column_family.cc:633] --------------- Options for column family [default]: 2024/11/19-15:14:31.434984 139852449619904 Options.comparator: leveldb.BytewiseComparator 2024/11/19-15:14:31.434991 139852449619904 Options.merge_operator: None 2024/11/19-15:14:31.434998 139852449619904 Options.compaction_filter: None 2024/11/19-15:14:31.435005 139852449619904 Options.compaction_filter_factory: Rdb_compact_filter_factory 2024/11/19-15:14:31.435012 139852449619904 Options.sst_partitioner_factory: SstPartitionerFixedPrefixEx 2024/11/19-15:14:31.435018 139852449619904 Options.memtable_factory: Dyna 2024/11/19-15:14:31.435028 139852449619904 Options.table_factory: DispatcherTable 2024/11/19-15:14:31.435033 139852449619904 table_factory options: {"default":"sng","readers":{"VecAutoSortTable":"auto_sort","BlockBasedTable":"bb","CSPPMemTabTable":"cspp_memtab_sst","ToplingZipTable":"dzip","SingleFastTable":"sng"},"level_writers":["sng","sng","light_dzip","dzip","dzip","dzip","dzip","dzip"],"//auto_compaction_max_wamp":10,"//trivial_move_always_max_output_level":4} 2024/11/19-15:14:31.435040 139852449619904 Options.write_buffer_size: 536870912 2024/11/19-15:14:31.435046 139852449619904 Options.max_write_buffer_number: 2 2024/11/19-15:14:31.435054 139852449619904 Options.compression: Snappy 2024/11/19-15:14:31.435063 139852449619904 Options.bottommost_compression: Disabled 2024/11/19-15:14:31.435070 139852449619904 Options.prefix_extractor: nullptr 2024/11/19-15:14:31.435074 139852449619904 Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024/11/19-15:14:31.435080 139852449619904 Options.num_levels: 7 2024/11/19-15:14:31.435087 139852449619904 Options.min_write_buffer_number_to_merge: 1 2024/11/19-15:14:31.435093 139852449619904 Options.max_write_buffer_number_to_maintain: 0 2024/11/19-15:14:31.435100 139852449619904 Options.max_write_buffer_size_to_maintain: 268435456 2024/11/19-15:14:31.435106 139852449619904 Options.bottommost_compression_opts.window_bits: -14 2024/11/19-15:14:31.435111 139852449619904 Options.bottommost_compression_opts.level: 32767 2024/11/19-15:14:31.435120 139852449619904 Options.bottommost_compression_opts.strategy: 0 2024/11/19-15:14:31.435126 139852449619904 Options.bottommost_compression_opts.max_dict_bytes: 0 2024/11/19-15:14:31.435133 139852449619904 Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024/11/19-15:14:31.435139 139852449619904 Options.bottommost_compression_opts.parallel_threads: 1 2024/11/19-15:14:31.435146 139852449619904 Options.bottommost_compression_opts.enabled: false 2024/11/19-15:14:31.435150 139852449619904 Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024/11/19-15:14:31.435156 139852449619904 Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024/11/19-15:14:31.435163 139852449619904 Options.compression_opts.window_bits: -14 2024/11/19-15:14:31.435170 139852449619904 Options.compression_opts.level: 32767 2024/11/19-15:14:31.435176 139852449619904 Options.compression_opts.strategy: 0 2024/11/19-15:14:31.435183 139852449619904 Options.compression_opts.max_dict_bytes: 0 2024/11/19-15:14:31.435187 139852449619904 Options.compression_opts.zstd_max_train_bytes: 0 2024/11/19-15:14:31.435193 139852449619904 Options.compression_opts.use_zstd_dict_trainer: true 2024/11/19-15:14:31.435200 139852449619904 Options.compression_opts.parallel_threads: 1 2024/11/19-15:14:31.435206 139852449619904 Options.compression_opts.enabled: false 2024/11/19-15:14:31.435213 139852449619904 Options.compression_opts.max_dict_buffer_bytes: 0 2024/11/19-15:14:31.435219 139852449619904 Options.level0_file_num_compaction_trigger: 4 2024/11/19-15:14:31.435223 139852449619904 Options.level0_slowdown_writes_trigger: 5 2024/11/19-15:14:31.435230 139852449619904 Options.level0_stop_writes_trigger: 10 2024/11/19-15:14:31.435236 139852449619904 Options.target_file_size_base: 33554432 2024/11/19-15:14:31.435242 139852449619904 Options.target_file_size_multiplier: 2 2024/11/19-15:14:31.435249 139852449619904 Options.max_bytes_for_level_base: 536870912 2024/11/19-15:14:31.435256 139852449619904 Options.level_compaction_dynamic_level_bytes: 0 2024/11/19-15:14:31.435260 139852449619904 Options.max_bytes_for_level_multiplier: 1.000000 2024/11/19-15:14:31.435268 139852449619904 Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024/11/19-15:14:31.435275 139852449619904 Options.max_bytes_for_level_multiplier_addtl[1]: 3 2024/11/19-15:14:31.435283 139852449619904 Options.max_bytes_for_level_multiplier_addtl[2]: 5 2024/11/19-15:14:31.435292 139852449619904 Options.max_bytes_for_level_multiplier_addtl[3]: 7 2024/11/19-15:14:31.435298 139852449619904 Options.max_bytes_for_level_multiplier_addtl[4]: 10 2024/11/19-15:14:31.435305 139852449619904 Options.max_bytes_for_level_multiplier_addtl[5]: 10 2024/11/19-15:14:31.435309 139852449619904 Options.max_bytes_for_level_multiplier_addtl[6]: 10 2024/11/19-15:14:31.435316 139852449619904 Options.max_bytes_for_level_multiplier_addtl[7]: 10 2024/11/19-15:14:31.435322 139852449619904 Options.max_sequential_skip_in_iterations: 8 2024/11/19-15:14:31.435328 139852449619904 Options.max_compaction_bytes: 4294967296 2024/11/19-15:14:31.435335 139852449619904 Options.arena_block_size: 1048576 2024/11/19-15:14:31.435341 139852449619904 Options.soft_pending_compaction_bytes_limit: 137438953472 2024/11/19-15:14:31.435346 139852449619904 Options.hard_pending_compaction_bytes_limit: 274877906944 2024/11/19-15:14:31.435352 139852449619904 Options.disable_auto_compactions: 1 2024/11/19-15:14:31.435359 139852449619904 Options.compaction_style: kCompactionStyleLevel 2024/11/19-15:14:31.435366 139852449619904 Options.compaction_pri: kMinOverlappingRatio 2024/11/19-15:14:31.435373 139852449619904 Options.compaction_options_universal.size_ratio: 1 2024/11/19-15:14:31.435382 139852449619904 Options.compaction_options_universal.min_merge_width: 2 2024/11/19-15:14:31.435386 139852449619904 Options.compaction_options_universal.max_merge_width: 4294967295 2024/11/19-15:14:31.435392 139852449619904 Options.compaction_options_universal.max_size_amplification_percent: 200 2024/11/19-15:14:31.435399 139852449619904 Options.compaction_options_universal.compression_size_percent: -1 2024/11/19-15:14:31.435406 139852449619904 Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024/11/19-15:14:31.435412 139852449619904 Options.compaction_options_fifo.max_table_files_size: 1073741824 2024/11/19-15:14:31.435419 139852449619904 Options.compaction_options_fifo.allow_compaction: 0 2024/11/19-15:14:31.435429 139852449619904 Options.table_properties_collectors: Rdb_tbl_prop_coll_factory; 2024/11/19-15:14:31.435433 139852449619904 Options.inplace_update_support: 0 2024/11/19-15:14:31.435440 139852449619904 Options.inplace_update_num_locks: 10000 2024/11/19-15:14:31.435447 139852449619904 Options.memtable_prefix_bloom_size_ratio: 0.000000 2024/11/19-15:14:31.435454 139852449619904 Options.allow_merge_memtables: 0 2024/11/19-15:14:31.435460 139852449619904 Options.memtable_whole_key_filtering: 0 2024/11/19-15:14:31.435466 139852449619904 Options.memtable_huge_page_size: 0 2024/11/19-15:14:31.435471 139852449619904 Options.bloom_locality: 0 2024/11/19-15:14:31.435477 139852449619904 Options.max_successive_merges: 0 2024/11/19-15:14:31.435484 139852449619904 Options.optimize_filters_for_hits: 0 2024/11/19-15:14:31.435490 139852449619904 Options.paranoid_file_checks: 0 2024/11/19-15:14:31.435497 139852449619904 Options.force_consistency_checks: 1 2024/11/19-15:14:31.435503 139852449619904 Options.report_bg_io_stats: 0 2024/11/19-15:14:31.435507 139852449619904 Options.ttl: 0 2024/11/19-15:14:31.435514 139852449619904 Options.periodic_compaction_seconds: 0 2024/11/19-15:14:31.435522 139852449619904 Options.preclude_last_level_data_seconds: 0 2024/11/19-15:14:31.435528 139852449619904 Options.preserve_internal_time_seconds: 0 2024/11/19-15:14:31.435536 139852449619904 Options.enable_blob_files: false 2024/11/19-15:14:31.435543 139852449619904 Options.min_blob_size: 0 2024/11/19-15:14:31.435550 139852449619904 Options.blob_file_size: 268435456 2024/11/19-15:14:31.435554 139852449619904 Options.blob_compression_type: NoCompression 2024/11/19-15:14:31.435561 139852449619904 Options.enable_blob_garbage_collection: false 2024/11/19-15:14:31.435567 139852449619904 Options.blob_garbage_collection_age_cutoff: 0.250000 2024/11/19-15:14:31.435574 139852449619904 Options.blob_garbage_collection_force_threshold: 1.000000 2024/11/19-15:14:31.435581 139852449619904 Options.blob_compaction_readahead_size: 0 2024/11/19-15:14:31.435587 139852449619904 Options.blob_file_starting_level: 0 2024/11/19-15:14:31.435592 139852449619904 Options.min_filter_level: 2 2024/11/19-15:14:31.435598 139852449619904 Options.experimental_mempurge_threshold: 0.000000 2024/11/19-15:14:31.435605 139852449619904 Options.memtable_max_range_deletions: 0 2024/11/19-15:14:31.437335 139852449619904 [column_family.cc:633] --------------- Options for column family [__system__]: 2024/11/19-15:14:31.437350 139852449619904 Options.comparator: leveldb.BytewiseComparator 2024/11/19-15:14:31.437356 139852449619904 Options.merge_operator: Rdb_system_merge_op 2024/11/19-15:14:31.437360 139852449619904 Options.compaction_filter: None 2024/11/19-15:14:31.437365 139852449619904 Options.compaction_filter_factory: None 2024/11/19-15:14:31.437372 139852449619904 Options.sst_partitioner_factory: None 2024/11/19-15:14:31.437377 139852449619904 Options.memtable_factory: CSPPMemTabFactory 2024/11/19-15:14:31.437381 139852449619904 Options.table_factory: DispatcherTable 2024/11/19-15:14:31.437386 139852449619904 table_factory options: {"default":"sng","readers":{"VecAutoSortTable":"auto_sort","BlockBasedTable":"bb","CSPPMemTabTable":"cspp_memtab_sst","ToplingZipTable":"dzip","SingleFastTable":"sng"},"level_writers":["sng","sng","light_dzip","dzip","dzip","dzip","dzip","dzip"],"//auto_compaction_max_wamp":10,"//trivial_move_always_max_output_level":4} 2024/11/19-15:14:31.437391 139852449619904 Options.write_buffer_size: 8388608 2024/11/19-15:14:31.437395 139852449619904 Options.max_write_buffer_number: 2 2024/11/19-15:14:31.437400 139852449619904 Options.compression: Snappy 2024/11/19-15:14:31.437404 139852449619904 Options.bottommost_compression: Disabled 2024/11/19-15:14:31.437409 139852449619904 Options.prefix_extractor: nullptr 2024/11/19-15:14:31.437413 139852449619904 Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024/11/19-15:14:31.437417 139852449619904 Options.num_levels: 7 2024/11/19-15:14:31.437422 139852449619904 Options.min_write_buffer_number_to_merge: 1 2024/11/19-15:14:31.437426 139852449619904 Options.max_write_buffer_number_to_maintain: 0 2024/11/19-15:14:31.437430 139852449619904 Options.max_write_buffer_size_to_maintain: 16777216 2024/11/19-15:14:31.437439 139852449619904 Options.bottommost_compression_opts.window_bits: -14 2024/11/19-15:14:31.437443 139852449619904 Options.bottommost_compression_opts.level: 32767 2024/11/19-15:14:31.437450 139852449619904 Options.bottommost_compression_opts.strategy: 0 2024/11/19-15:14:31.437456 139852449619904 Options.bottommost_compression_opts.max_dict_bytes: 0 2024/11/19-15:14:31.437463 139852449619904 Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024/11/19-15:14:31.437467 139852449619904 Options.bottommost_compression_opts.parallel_threads: 1 2024/11/19-15:14:31.437474 139852449619904 Options.bottommost_compression_opts.enabled: false 2024/11/19-15:14:31.437481 139852449619904 Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024/11/19-15:14:31.437487 139852449619904 Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024/11/19-15:14:31.437494 139852449619904 Options.compression_opts.window_bits: -14 2024/11/19-15:14:31.437500 139852449619904 Options.compression_opts.level: 32767 2024/11/19-15:14:31.437505 139852449619904 Options.compression_opts.strategy: 0 2024/11/19-15:14:31.437511 139852449619904 Options.compression_opts.max_dict_bytes: 0 2024/11/19-15:14:31.437517 139852449619904 Options.compression_opts.zstd_max_train_bytes: 0 2024/11/19-15:14:31.437524 139852449619904 Options.compression_opts.use_zstd_dict_trainer: true 2024/11/19-15:14:31.437533 139852449619904 Options.compression_opts.parallel_threads: 1 2024/11/19-15:14:31.437539 139852449619904 Options.compression_opts.enabled: false 2024/11/19-15:14:31.437543 139852449619904 Options.compression_opts.max_dict_buffer_bytes: 0 2024/11/19-15:14:31.437550 139852449619904 Options.level0_file_num_compaction_trigger: 4 2024/11/19-15:14:31.437556 139852449619904 Options.level0_slowdown_writes_trigger: 5 2024/11/19-15:14:31.437562 139852449619904 Options.level0_stop_writes_trigger: 10 2024/11/19-15:14:31.437569 139852449619904 Options.target_file_size_base: 8388608 2024/11/19-15:14:31.437576 139852449619904 Options.target_file_size_multiplier: 2 2024/11/19-15:14:31.437580 139852449619904 Options.max_bytes_for_level_base: 8388608 2024/11/19-15:14:31.437586 139852449619904 Options.level_compaction_dynamic_level_bytes: 0 2024/11/19-15:14:31.437595 139852449619904 Options.max_bytes_for_level_multiplier: 1.000000 2024/11/19-15:14:31.437603 139852449619904 Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024/11/19-15:14:31.437609 139852449619904 Options.max_bytes_for_level_multiplier_addtl[1]: 3 2024/11/19-15:14:31.437616 139852449619904 Options.max_bytes_for_level_multiplier_addtl[2]: 5 2024/11/19-15:14:31.437622 139852449619904 Options.max_bytes_for_level_multiplier_addtl[3]: 7 2024/11/19-15:14:31.437626 139852449619904 Options.max_bytes_for_level_multiplier_addtl[4]: 10 2024/11/19-15:14:31.437633 139852449619904 Options.max_bytes_for_level_multiplier_addtl[5]: 10 2024/11/19-15:14:31.437639 139852449619904 Options.max_bytes_for_level_multiplier_addtl[6]: 10 2024/11/19-15:14:31.437645 139852449619904 Options.max_bytes_for_level_multiplier_addtl[7]: 10 2024/11/19-15:14:31.437652 139852449619904 Options.max_sequential_skip_in_iterations: 8 2024/11/19-15:14:31.437667 139852449619904 Options.max_compaction_bytes: 4294967296 2024/11/19-15:14:31.437672 139852449619904 Options.arena_block_size: 1048576 2024/11/19-15:14:31.437678 139852449619904 Options.soft_pending_compaction_bytes_limit: 68719476736 2024/11/19-15:14:31.437687 139852449619904 Options.hard_pending_compaction_bytes_limit: 274877906944 2024/11/19-15:14:31.437694 139852449619904 Options.disable_auto_compactions: 1 2024/11/19-15:14:31.437701 139852449619904 Options.compaction_style: kCompactionStyleLevel 2024/11/19-15:14:31.437708 139852449619904 Options.compaction_pri: kMinOverlappingRatio 2024/11/19-15:14:31.437713 139852449619904 Options.compaction_options_universal.size_ratio: 1 2024/11/19-15:14:31.437719 139852449619904 Options.compaction_options_universal.min_merge_width: 2 2024/11/19-15:14:31.437726 139852449619904 Options.compaction_options_universal.max_merge_width: 4294967295 2024/11/19-15:14:31.437733 139852449619904 Options.compaction_options_universal.max_size_amplification_percent: 200 2024/11/19-15:14:31.437739 139852449619904 Options.compaction_options_universal.compression_size_percent: -1 2024/11/19-15:14:31.437746 139852449619904 Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024/11/19-15:14:31.437750 139852449619904 Options.compaction_options_fifo.max_table_files_size: 1073741824 2024/11/19-15:14:31.437757 139852449619904 Options.compaction_options_fifo.allow_compaction: 0 2024/11/19-15:14:31.437765 139852449619904 Options.table_properties_collectors: 2024/11/19-15:14:31.437774 139852449619904 Options.inplace_update_support: 0 2024/11/19-15:14:31.437781 139852449619904 Options.inplace_update_num_locks: 10000 2024/11/19-15:14:31.437788 139852449619904 Options.memtable_prefix_bloom_size_ratio: 0.000000 2024/11/19-15:14:31.437792 139852449619904 Options.allow_merge_memtables: 0 2024/11/19-15:14:31.437799 139852449619904 Options.memtable_whole_key_filtering: 0 2024/11/19-15:14:31.437805 139852449619904 Options.memtable_huge_page_size: 0 2024/11/19-15:14:31.437812 139852449619904 Options.bloom_locality: 0 2024/11/19-15:14:31.437818 139852449619904 Options.max_successive_merges: 0 2024/11/19-15:14:31.437825 139852449619904 Options.optimize_filters_for_hits: 0 2024/11/19-15:14:31.437829 139852449619904 Options.paranoid_file_checks: 0 2024/11/19-15:14:31.437835 139852449619904 Options.force_consistency_checks: 1 2024/11/19-15:14:31.437842 139852449619904 Options.report_bg_io_stats: 0 2024/11/19-15:14:31.437848 139852449619904 Options.ttl: 0 2024/11/19-15:14:31.437854 139852449619904 Options.periodic_compaction_seconds: 0 2024/11/19-15:14:31.437861 139852449619904 Options.preclude_last_level_data_seconds: 0 2024/11/19-15:14:31.437865 139852449619904 Options.preserve_internal_time_seconds: 0 2024/11/19-15:14:31.437872 139852449619904 Options.enable_blob_files: false 2024/11/19-15:14:31.437878 139852449619904 Options.min_blob_size: 0 2024/11/19-15:14:31.437884 139852449619904 Options.blob_file_size: 268435456 2024/11/19-15:14:31.437891 139852449619904 Options.blob_compression_type: NoCompression 2024/11/19-15:14:31.437898 139852449619904 Options.enable_blob_garbage_collection: false 2024/11/19-15:14:31.437904 139852449619904 Options.blob_garbage_collection_age_cutoff: 0.250000 2024/11/19-15:14:31.437909 139852449619904 Options.blob_garbage_collection_force_threshold: 1.000000 2024/11/19-15:14:31.437916 139852449619904 Options.blob_compaction_readahead_size: 0 2024/11/19-15:14:31.437940 139852449619904 Options.blob_file_starting_level: 0 2024/11/19-15:14:31.437947 139852449619904 Options.min_filter_level: 2 2024/11/19-15:14:31.437955 139852449619904 Options.experimental_mempurge_threshold: 0.000000 2024/11/19-15:14:31.437962 139852449619904 Options.memtable_max_range_deletions: 0 2024/11/19-15:14:31.540679 139850128819968 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/19-15:14:31.540729 139850128819968 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/19-15:14:31.540755 139850128819968 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/19-15:14:31.540773 139850128819968 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/19-15:14:31.540791 139850128819968 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/19-15:14:31.577282 139852449619904 [version_set.cc:6254] Recovered from manifest file:/storage/mytopling/.rocksdb/MANIFEST-004299 succeeded,manifest_file_number is 4299, next_file_number is 4319, last_sequence is 105631341, log_number is 4316,prev_log_number is 0,max_column_family is 1,min_log_number_to_keep is 4316 2024/11/19-15:14:31.577311 139852449619904 [version_set.cc:6269] Column family [default] (ID 0), log number is 4295 2024/11/19-15:14:31.577319 139852449619904 [version_set.cc:6269] Column family [__system__] (ID 1), log number is 4316 2024/11/19-15:14:31.577353 139852449619904 [db_impl_open.cc:639] DB ID: d89b4006-2803-486e-b7a2-759bcf348921 2024/11/19-15:14:31.578349 139852449619904 EVENT_LOG_v1 {"time_micros": 1732000471578338, "job": 1, "event": "recovery_started", "wal_files": [4316]} 2024/11/19-15:14:31.578366 139852449619904 [db_impl_open.cc:1132] Recovering log #4316 mode 2 2024/11/19-15:14:31.578429 139852449619904 EVENT_LOG_v1 {"time_micros": 1732000471578428, "job": 1, "event": "recovery_finished"} 2024/11/19-15:14:31.578553 139852449619904 [version_set.cc:5724] Creating manifest 4321 2024/11/19-15:14:31.579722 139852449619904 [db_impl_open.cc:2123] SstFileManager instance 0x170b5b60 2024/11/19-15:14:31.580533 139852449619904 DB pointer 0x170f1380 2024/11/19-15:14:31.580600 139852449619904 [WARN] [pessimistic_transaction_db.cc:310] Transaction write_policy is WRITE_COMMITTED 2024/11/19-15:14:31.585899 139852449619904 [db_impl.cc:1284] SetOptions() on column family [default], inputs: 2024/11/19-15:14:31.585927 139852449619904 [db_impl.cc:1288] disable_auto_compactions: false 2024/11/19-15:14:31.585935 139852449619904 [db_impl.cc:1292] [default] SetOptions() succeeded 2024/11/19-15:14:31.585941 139852449619904 [cf_options.cc:1051] write_buffer_size: 536870912 2024/11/19-15:14:31.585948 139852449619904 [cf_options.cc:1054] max_write_buffer_number: 2 2024/11/19-15:14:31.585953 139852449619904 [cf_options.cc:1056] arena_block_size: 1048576 2024/11/19-15:14:31.585959 139852449619904 [cf_options.cc:1059] memtable_prefix_bloom_ratio: 0.000000 2024/11/19-15:14:31.585965 139852449619904 [cf_options.cc:1062] allow_merge_memtables: 0 2024/11/19-15:14:31.585970 139852449619904 [cf_options.cc:1064] memtable_whole_key_filtering: 0 2024/11/19-15:14:31.585975 139852449619904 [cf_options.cc:1066] memtable_huge_page_size: 0 2024/11/19-15:14:31.585980 139852449619904 [cf_options.cc:1069] max_successive_merges: 0 2024/11/19-15:14:31.585986 139852449619904 [cf_options.cc:1072] inplace_update_num_locks: 10000 2024/11/19-15:14:31.585991 139852449619904 [cf_options.cc:1075] prefix_extractor: nullptr 2024/11/19-15:14:31.585996 139852449619904 [cf_options.cc:1079] disable_auto_compactions: 0 2024/11/19-15:14:31.586001 139852449619904 [cf_options.cc:1081] soft_pending_compaction_bytes_limit: 137438953472 2024/11/19-15:14:31.586007 139852449619904 [cf_options.cc:1083] hard_pending_compaction_bytes_limit: 274877906944 2024/11/19-15:14:31.586012 139852449619904 [cf_options.cc:1085] level0_file_num_compaction_trigger: 4 2024/11/19-15:14:31.586021 139852449619904 [cf_options.cc:1087] level0_slowdown_writes_trigger: 5 2024/11/19-15:14:31.586026 139852449619904 [cf_options.cc:1089] level0_stop_writes_trigger: 10 2024/11/19-15:14:31.586033 139852449619904 [cf_options.cc:1091] max_compaction_bytes: 4294967296 2024/11/19-15:14:31.586038 139852449619904 [cf_options.cc:1093] target_file_size_base: 33554432 2024/11/19-15:14:31.586045 139852449619904 [cf_options.cc:1095] target_file_size_multiplier: 2 2024/11/19-15:14:31.586052 139852449619904 [cf_options.cc:1097] max_bytes_for_level_base: 536870912 2024/11/19-15:14:31.586057 139852449619904 [cf_options.cc:1099] max_bytes_for_level_multiplier: 1.000000 2024/11/19-15:14:31.586064 139852449619904 [cf_options.cc:1101] ttl: 0 2024/11/19-15:14:31.586072 139852449619904 [cf_options.cc:1103] periodic_compaction_seconds: 0 2024/11/19-15:14:31.586080 139852449619904 [cf_options.cc:1117] max_bytes_for_level_multiplier_additional: 1, 3, 5, 7, 10, 10, 10, 10 2024/11/19-15:14:31.586087 139852449619904 [cf_options.cc:1119] max_sequential_skip_in_iterations: 8 2024/11/19-15:14:31.586091 139852449619904 [cf_options.cc:1121] check_flush_compaction_key_order: 0 2024/11/19-15:14:31.586098 139852449619904 [cf_options.cc:1123] paranoid_file_checks: 0 2024/11/19-15:14:31.586104 139852449619904 [cf_options.cc:1125] report_bg_io_stats: 0 2024/11/19-15:14:31.586111 139852449619904 [cf_options.cc:1127] compression: 1 2024/11/19-15:14:31.586122 139852449619904 [cf_options.cc:1129] experimental_mempurge_threshold: 0.000000 2024/11/19-15:14:31.586129 139852449619904 [cf_options.cc:1134] compaction_options_universal.size_ratio : 1 2024/11/19-15:14:31.586133 139852449619904 [cf_options.cc:1136] compaction_options_universal.min_merge_width : 2 2024/11/19-15:14:31.586140 139852449619904 [cf_options.cc:1138] compaction_options_universal.max_merge_width : -1 2024/11/19-15:14:31.586146 139852449619904 [cf_options.cc:1140] compaction_options_universal.max_size_amplification_percent : 200 2024/11/19-15:14:31.586153 139852449619904 [cf_options.cc:1143] compaction_options_universal.compression_size_percent : -1 2024/11/19-15:14:31.586160 139852449619904 [cf_options.cc:1146] compaction_options_universal.stop_style : 1 2024/11/19-15:14:31.586167 139852449619904 [cf_options.cc:1148] compaction_options_universal.allow_trivial_move : 0 2024/11/19-15:14:31.586171 139852449619904 [cf_options.cc:1151] compaction_options_universal.incremental : 0 2024/11/19-15:14:31.586178 139852449619904 [cf_options.cc:1155] compaction_options_fifo.max_table_files_size : 1073741824 2024/11/19-15:14:31.586185 139852449619904 [cf_options.cc:1157] compaction_options_fifo.allow_compaction : 0 2024/11/19-15:14:31.586191 139852449619904 [cf_options.cc:1161] enable_blob_files: false 2024/11/19-15:14:31.586198 139852449619904 [cf_options.cc:1163] min_blob_size: 0 2024/11/19-15:14:31.586205 139852449619904 [cf_options.cc:1165] blob_file_size: 268435456 2024/11/19-15:14:31.586209 139852449619904 [cf_options.cc:1167] blob_compression_type: NoCompression 2024/11/19-15:14:31.586216 139852449619904 [cf_options.cc:1169] enable_blob_garbage_collection: false 2024/11/19-15:14:31.586223 139852449619904 [cf_options.cc:1171] blob_garbage_collection_age_cutoff: 0.250000 2024/11/19-15:14:31.586224 139850978166528 [compaction_job.cc:2437] [default] [JOB 3] Compacting 4@1 files to L2, score 5.14, subcompactions 1 : 1 2024/11/19-15:14:31.586230 139852449619904 [cf_options.cc:1173] blob_garbage_collection_force_threshold: 1.000000 2024/11/19-15:14:31.586237 139852449619904 [cf_options.cc:1175] blob_compaction_readahead_size: 0 2024/11/19-15:14:31.586254 139852449619904 [cf_options.cc:1177] blob_file_starting_level: 0 2024/11/19-15:14:31.586273 139851179493120 [compaction_job.cc:2437] [default] [JOB 4] Compacting 3@1 files to L2, score 4.89, subcompactions 1 : 1 2024/11/19-15:14:31.586275 139852449619904 [cf_options.cc:1179] prepopulate_blob_cache: disable 2024/11/19-15:14:31.586276 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 6 Base level 1, inputs: [4227(31MB) 4229(31MB) 4235(31MB) 4239(31MB)] 2024/11/19-15:14:31.586284 139852449619904 [cf_options.cc:1183] last_level_temperature: 0 2024/11/19-15:14:31.586314 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 6 Base level 1, inputs: [4243(26MB) 4149(31MB) 4154(11MB)] 2024/11/19-15:14:31.586323 139851169007360 [compaction_job.cc:2437] [default] [JOB 5] Compacting 4@1 files to L2, score 4.75, subcompactions 1 : 1 2024/11/19-15:14:31.586355 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 6 Base level 1, inputs: [4157(31MB) 4160(31MB) 4162(31MB) 4164(31MB)] 2024/11/19-15:14:31.586376 139851158521600 [compaction_job.cc:2437] [default] [JOB 6] Compacting 4@1 files to L2, score 4.50, subcompactions 1 : 1 2024/11/19-15:14:31.586416 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 6 Base level 1, inputs: [4168(31MB) 4170(31MB) 4173(31MB) 4175(31MB)] 2024/11/19-15:14:31.586552 139851137550080 [compaction_job.cc:2437] [default] [JOB 8] Compacting 4@1 files to L2, score 4.00, subcompactions 1 : 1 2024/11/19-15:14:31.586561 139851127064320 [compaction_job.cc:2437] [default] [JOB 9] Compacting 4@1 files to L2, score 3.75, subcompactions 1 : 1 2024/11/19-15:14:31.586562 139851148035840 [compaction_job.cc:2437] [default] [JOB 7] Compacting 4@1 files to L2, score 4.25, subcompactions 1 : 1 2024/11/19-15:14:31.586565 139852449619904 [db_impl.cc:1284] SetOptions() on column family [__system__], inputs: 2024/11/19-15:14:31.586598 139852449619904 [db_impl.cc:1288] disable_auto_compactions: false 2024/11/19-15:14:31.586615 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 6 Base level 1, inputs: [4182(31MB) 4183(31MB) 4185(31MB) 4186(31MB)] 2024/11/19-15:14:31.586633 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 6 Base level 1, inputs: [4187(31MB) 4188(31MB) 4190(31MB) 4191(31MB)] 2024/11/19-15:14:31.586645 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 6 Base level 1, inputs: [4176(31MB) 4178(31MB) 4179(31MB) 4181(31MB)] 2024/11/19-15:14:31.586651 139852449619904 [db_impl.cc:1292] [__system__] SetOptions() succeeded 2024/11/19-15:14:31.586671 139852449619904 [cf_options.cc:1051] write_buffer_size: 8388608 2024/11/19-15:14:31.586684 139852449619904 [cf_options.cc:1054] max_write_buffer_number: 2 2024/11/19-15:14:31.586702 139852449619904 [cf_options.cc:1056] arena_block_size: 1048576 2024/11/19-15:14:31.586710 139852449619904 [cf_options.cc:1059] memtable_prefix_bloom_ratio: 0.000000 2024/11/19-15:14:31.586723 139852449619904 [cf_options.cc:1062] allow_merge_memtables: 0 2024/11/19-15:14:31.586734 139852449619904 [cf_options.cc:1064] memtable_whole_key_filtering: 0 2024/11/19-15:14:31.586741 139852449619904 [cf_options.cc:1066] memtable_huge_page_size: 0 2024/11/19-15:14:31.586745 139852449619904 [cf_options.cc:1069] max_successive_merges: 0 2024/11/19-15:14:31.586752 139852449619904 [cf_options.cc:1072] inplace_update_num_locks: 10000 2024/11/19-15:14:31.586759 139852449619904 [cf_options.cc:1075] prefix_extractor: nullptr 2024/11/19-15:14:31.586766 139852449619904 [cf_options.cc:1079] disable_auto_compactions: 0 2024/11/19-15:14:31.586772 139852449619904 [cf_options.cc:1081] soft_pending_compaction_bytes_limit: 68719476736 2024/11/19-15:14:31.586782 139852449619904 [cf_options.cc:1083] hard_pending_compaction_bytes_limit: 274877906944 2024/11/19-15:14:31.586787 139852449619904 [cf_options.cc:1085] level0_file_num_compaction_trigger: 4 2024/11/19-15:14:31.586794 139852449619904 [cf_options.cc:1087] level0_slowdown_writes_trigger: 5 2024/11/19-15:14:31.586800 139852449619904 [cf_options.cc:1089] level0_stop_writes_trigger: 10 2024/11/19-15:14:31.586807 139852449619904 [cf_options.cc:1091] max_compaction_bytes: 4294967296 2024/11/19-15:14:31.586814 139852449619904 [cf_options.cc:1093] target_file_size_base: 8388608 2024/11/19-15:14:31.586820 139852449619904 [cf_options.cc:1095] target_file_size_multiplier: 2 2024/11/19-15:14:31.586825 139852449619904 [cf_options.cc:1097] max_bytes_for_level_base: 8388608 2024/11/19-15:14:31.586831 139852449619904 [cf_options.cc:1099] max_bytes_for_level_multiplier: 1.000000 2024/11/19-15:14:31.586838 139852449619904 [cf_options.cc:1101] ttl: 0 2024/11/19-15:14:31.586845 139852449619904 [cf_options.cc:1103] periodic_compaction_seconds: 0 2024/11/19-15:14:31.586853 139852449619904 [cf_options.cc:1117] max_bytes_for_level_multiplier_additional: 1, 3, 5, 7, 10, 10, 10, 10 2024/11/19-15:14:31.586862 139852449619904 [cf_options.cc:1119] max_sequential_skip_in_iterations: 8 2024/11/19-15:14:31.586871 139852449619904 [cf_options.cc:1121] check_flush_compaction_key_order: 1 2024/11/19-15:14:31.586878 139852449619904 [cf_options.cc:1123] paranoid_file_checks: 0 2024/11/19-15:14:31.586887 139852449619904 [cf_options.cc:1125] report_bg_io_stats: 0 2024/11/19-15:14:31.586897 139852449619904 [cf_options.cc:1127] compression: 1 2024/11/19-15:14:31.586906 139852449619904 [cf_options.cc:1129] experimental_mempurge_threshold: 0.000000 2024/11/19-15:14:31.586917 139852449619904 [cf_options.cc:1134] compaction_options_universal.size_ratio : 1 2024/11/19-15:14:31.586937 139852449619904 [cf_options.cc:1136] compaction_options_universal.min_merge_width : 2 2024/11/19-15:14:31.586943 139852449619904 [cf_options.cc:1138] compaction_options_universal.max_merge_width : -1 2024/11/19-15:14:31.586953 139852449619904 [cf_options.cc:1140] compaction_options_universal.max_size_amplification_percent : 200 2024/11/19-15:14:31.586963 139852449619904 [cf_options.cc:1143] compaction_options_universal.compression_size_percent : -1 2024/11/19-15:14:31.586973 139852449619904 [cf_options.cc:1146] compaction_options_universal.stop_style : 1 2024/11/19-15:14:31.586982 139852449619904 [cf_options.cc:1148] compaction_options_universal.allow_trivial_move : 0 2024/11/19-15:14:31.586992 139852449619904 [cf_options.cc:1151] compaction_options_universal.incremental : 0 2024/11/19-15:14:31.586998 139852449619904 [cf_options.cc:1155] compaction_options_fifo.max_table_files_size : 1073741824 2024/11/19-15:14:31.587008 139852449619904 [cf_options.cc:1157] compaction_options_fifo.allow_compaction : 0 2024/11/19-15:14:31.587018 139852449619904 [cf_options.cc:1161] enable_blob_files: false 2024/11/19-15:14:31.587026 139852449619904 [cf_options.cc:1163] min_blob_size: 0 2024/11/19-15:14:31.587036 139852449619904 [cf_options.cc:1165] blob_file_size: 268435456 2024/11/19-15:14:31.587044 139852449619904 [cf_options.cc:1167] blob_compression_type: NoCompression 2024/11/19-15:14:31.587054 139852449619904 [cf_options.cc:1169] enable_blob_garbage_collection: false 2024/11/19-15:14:31.587061 139852449619904 [cf_options.cc:1171] blob_garbage_collection_age_cutoff: 0.250000 2024/11/19-15:14:31.587072 139852449619904 [cf_options.cc:1173] blob_garbage_collection_force_threshold: 1.000000 2024/11/19-15:14:31.587083 139852449619904 [cf_options.cc:1175] blob_compaction_readahead_size: 0 2024/11/19-15:14:31.587093 139852449619904 [cf_options.cc:1177] blob_file_starting_level: 0 2024/11/19-15:14:31.587103 139852449619904 [cf_options.cc:1179] prepopulate_blob_cache: disable 2024/11/19-15:14:31.587113 139852449619904 [cf_options.cc:1183] last_level_temperature: 0 2024/11/19-15:14:31.874487 139851137550080 [dcompact_etcd.cc:1163] job-00008/att-00: init time wrt = 0.001, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.287 sec, input = 126.870 MiB 2024/11/19-15:14:31.892273 139851127064320 [dcompact_etcd.cc:1163] job-00009/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.305 sec, input = 126.908 MiB 2024/11/19-15:14:31.902416 139851158521600 [dcompact_etcd.cc:1163] job-00006/att-00: init time wrt = 0.001, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.315 sec, input = 126.925 MiB 2024/11/19-15:14:31.907413 139850978166528 [dcompact_etcd.cc:1163] job-00003/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.320 sec, input = 127.822 MiB 2024/11/19-15:14:31.915236 139851179493120 [dcompact_etcd.cc:1163] job-00004/att-00: init time wrt = 0.001, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.328 sec, input = 70.230 MiB 2024/11/19-15:14:31.918903 139851148035840 [dcompact_etcd.cc:1163] job-00007/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.332 sec, input = 126.998 MiB 2024/11/19-15:14:31.919786 139851169007360 [dcompact_etcd.cc:1163] job-00005/att-00: init time wrt = 0.001, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.333 sec, input = 126.886 MiB 2024/11/19-15:14:33.680358 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-00 2024/11/19-15:14:33.680443 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00004/att-00/compact.done) = No such file or directory: 1.756 sec on http probe failed: 1, retry 2024/11/19-15:14:35.056371 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-00 2024/11/19-15:14:35.056463 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00008/att-00/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:14:35.071372 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-00 2024/11/19-15:14:35.071441 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00009/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:14:35.080609 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-00 2024/11/19-15:14:35.080675 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00006/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:14:35.093085 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-00 2024/11/19-15:14:35.093136 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00004/att-00/compact.done) = No such file or directory: 3.170 sec on http probe failed: 2, retry 2024/11/19-15:14:35.099499 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-00 2024/11/19-15:14:35.099552 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00007/att-00/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:14:35.099770 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-00 2024/11/19-15:14:35.099825 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00005/att-00/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:14:35.111936 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-00 2024/11/19-15:14:35.111992 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00003/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:14:36.502965 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-00 2024/11/19-15:14:36.503031 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00004/att-00/compact.done) = No such file or directory: 4.583 sec on http probe failed: 3, retry 2024/11/19-15:14:37.062538 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-00 2024/11/19-15:14:37.062659 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00008/att-00/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:14:37.079167 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-00 2024/11/19-15:14:37.079225 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00009/att-00/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:14:37.084935 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-00 2024/11/19-15:14:37.084991 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00006/att-00/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:14:37.106274 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-00 2024/11/19-15:14:37.106307 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-00 2024/11/19-15:14:37.106322 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00005/att-00/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:14:37.106357 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00007/att-00/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:14:37.119590 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-00 2024/11/19-15:14:37.119638 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00003/att-00/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:14:37.912823 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-00 2024/11/19-15:14:37.912900 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00004/att-00/compact.done) = No such file or directory: 5.992 sec on http probe failed: 4, retry 2024/11/19-15:14:39.072486 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-00 2024/11/19-15:14:39.072575 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00008/att-00/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:14:39.088690 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-00 2024/11/19-15:14:39.088750 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00009/att-00/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:14:39.090873 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-00 2024/11/19-15:14:39.090939 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00006/att-00/compact.done) = No such file or directory: 7.183 sec on http probe failed: 3, retry 2024/11/19-15:14:39.110847 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-00 2024/11/19-15:14:39.110899 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00005/att-00/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:14:39.113617 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-00 2024/11/19-15:14:39.113667 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00007/att-00/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:14:39.127210 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-00 2024/11/19-15:14:39.127263 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00003/att-00/compact.done) = No such file or directory: 7.212 sec on http probe failed: 3, retry 2024/11/19-15:14:39.322811 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-00 2024/11/19-15:14:39.322893 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00004/att-00/compact.done) = No such file or directory: 7.402 sec on http probe failed: 5, giveup 2024/11/19-15:14:39.322926 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00004/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 70.230 sec, real 7.736 (submit 328.178 ms wait 7.402 s), input 70.230 MiB, send shutdown 2024/11/19-15:14:39.331155 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-00 2024/11/19-15:14:39.331373 139851179493120 [dcompact_etcd.cc:1773] job-00004/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:14:39.610603 139851179493120 [dcompact_etcd.cc:1163] job-00004/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.279 sec, input = 70.230 MiB 2024/11/19-15:14:41.082210 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-00 2024/11/19-15:14:41.082300 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00008/att-00/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:14:41.094461 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-00 2024/11/19-15:14:41.094527 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00009/att-00/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:14:41.095267 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-00 2024/11/19-15:14:41.095326 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00006/att-00/compact.done) = No such file or directory: 9.189 sec on http probe failed: 4, retry 2024/11/19-15:14:41.119213 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-00 2024/11/19-15:14:41.119262 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00005/att-00/compact.done) = No such file or directory: 9.191 sec on http probe failed: 4, retry 2024/11/19-15:14:41.121895 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-00 2024/11/19-15:14:41.121954 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00007/att-00/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:14:41.134902 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-00 2024/11/19-15:14:41.134957 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00003/att-00/compact.done) = No such file or directory: 9.220 sec on http probe failed: 4, retry 2024/11/19-15:14:41.376195 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-01 2024/11/19-15:14:41.376271 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00004/att-01/compact.done) = No such file or directory: 1.756 sec on http probe failed: 1, retry 2024/11/19-15:14:42.789561 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-01 2024/11/19-15:14:42.789634 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00004/att-01/compact.done) = No such file or directory: 3.170 sec on http probe failed: 2, retry 2024/11/19-15:14:43.091730 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-00 2024/11/19-15:14:43.091797 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00008/att-00/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:14:43.091815 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00008/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.870 sec, real 11.504 (submit 286.982 ms wait 11.208 s), input 126.870 MiB, send shutdown 2024/11/19-15:14:43.096266 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-00 2024/11/19-15:14:43.096479 139851137550080 [dcompact_etcd.cc:1773] job-00008/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:14:43.103444 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-00 2024/11/19-15:14:43.103496 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00009/att-00/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:14:43.103510 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00009/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.908 sec, real 11.516 (submit 305.033 ms wait 11.202 s), input 126.908 MiB, send shutdown 2024/11/19-15:14:43.103603 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-00 2024/11/19-15:14:43.103657 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00006/att-00/compact.done) = No such file or directory: 11.193 sec on http probe failed: 5, giveup 2024/11/19-15:14:43.103672 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00006/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.925 sec, real 11.516 (submit 315.201 ms wait 11.193 s), input 126.925 MiB, send shutdown 2024/11/19-15:14:43.111275 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-00 2024/11/19-15:14:43.111462 139851127064320 [dcompact_etcd.cc:1773] job-00009/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:14:43.112330 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-00 2024/11/19-15:14:43.112508 139851158521600 [dcompact_etcd.cc:1773] job-00006/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:14:43.126846 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-00 2024/11/19-15:14:43.126895 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00007/att-00/compact.done) = No such file or directory: 11.203 sec on http probe failed: 5, giveup 2024/11/19-15:14:43.126909 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00007/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.998 sec, real 11.540 (submit 331.587 ms wait 11.203 s), input 126.998 MiB, send shutdown 2024/11/19-15:14:43.128192 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-00 2024/11/19-15:14:43.128242 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00005/att-00/compact.done) = No such file or directory: 11.200 sec on http probe failed: 5, giveup 2024/11/19-15:14:43.128257 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00005/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.886 sec, real 11.541 (submit 332.620 ms wait 11.200 s), input 126.886 MiB, send shutdown 2024/11/19-15:14:43.131305 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-00 2024/11/19-15:14:43.131471 139851148035840 [dcompact_etcd.cc:1773] job-00007/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:14:43.137044 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-00 2024/11/19-15:14:43.137215 139851169007360 [dcompact_etcd.cc:1773] job-00005/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:14:43.143227 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-00 2024/11/19-15:14:43.143274 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00003/att-00/compact.done) = No such file or directory: 11.228 sec on http probe failed: 5, giveup 2024/11/19-15:14:43.143288 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00003/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.822 sec, real 11.556 (submit 320.370 ms wait 11.228 s), input 127.822 MiB, send shutdown 2024/11/19-15:14:43.150738 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-00 2024/11/19-15:14:43.150903 139850978166528 [dcompact_etcd.cc:1773] job-00003/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:14:43.392579 139851137550080 [dcompact_etcd.cc:1163] job-00008/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.296 sec, input = 126.870 MiB 2024/11/19-15:14:43.408571 139851127064320 [dcompact_etcd.cc:1163] job-00009/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.297 sec, input = 126.908 MiB 2024/11/19-15:14:43.423993 139851158521600 [dcompact_etcd.cc:1163] job-00006/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.311 sec, input = 126.925 MiB 2024/11/19-15:14:43.423993 139851148035840 [dcompact_etcd.cc:1163] job-00007/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.292 sec, input = 126.998 MiB 2024/11/19-15:14:43.442022 139851169007360 [dcompact_etcd.cc:1163] job-00005/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.305 sec, input = 126.886 MiB 2024/11/19-15:14:43.451627 139850978166528 [dcompact_etcd.cc:1163] job-00003/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.301 sec, input = 127.822 MiB 2024/11/19-15:14:44.203947 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-01 2024/11/19-15:14:44.204020 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00004/att-01/compact.done) = No such file or directory: 4.584 sec on http probe failed: 3, retry 2024/11/19-15:14:45.613693 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-01 2024/11/19-15:14:45.613765 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00004/att-01/compact.done) = No such file or directory: 5.998 sec on http probe failed: 4, retry 2024/11/19-15:14:46.573075 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-01 2024/11/19-15:14:46.573150 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00008/att-01/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:14:46.589543 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-01 2024/11/19-15:14:46.589606 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00009/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:14:46.607879 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-01 2024/11/19-15:14:46.607950 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00007/att-01/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:14:46.609042 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-01 2024/11/19-15:14:46.609093 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00006/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:14:46.620019 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-01 2024/11/19-15:14:46.620075 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00005/att-01/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:14:46.651506 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-01 2024/11/19-15:14:46.651557 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00003/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:14:47.027639 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-01 2024/11/19-15:14:47.027702 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00004/att-01/compact.done) = No such file or directory: 7.408 sec on http probe failed: 5, giveup 2024/11/19-15:14:47.027720 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00004/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 70.230 sec, real 7.696 (submit 279.136 ms wait 7.408 s), input 70.230 MiB, send shutdown 2024/11/19-15:14:47.035678 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-01 2024/11/19-15:14:47.035897 139851179493120 [dcompact_etcd.cc:1773] job-00004/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:14:47.306459 139851179493120 [dcompact_etcd.cc:1163] job-00004/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.270 sec, input = 70.230 MiB 2024/11/19-15:14:48.581837 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-01 2024/11/19-15:14:48.581962 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00008/att-01/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:14:48.594575 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-01 2024/11/19-15:14:48.594628 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00009/att-01/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:14:48.614403 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-01 2024/11/19-15:14:48.614457 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00006/att-01/compact.done) = No such file or directory: 5.185 sec on http probe failed: 2, retry 2024/11/19-15:14:48.617249 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-01 2024/11/19-15:14:48.617383 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00007/att-01/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:14:48.627606 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-01 2024/11/19-15:14:48.627659 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00005/att-01/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:14:48.661771 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-01 2024/11/19-15:14:48.661819 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00003/att-01/compact.done) = No such file or directory: 5.200 sec on http probe failed: 2, retry 2024/11/19-15:14:49.068165 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-02 2024/11/19-15:14:49.068242 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00004/att-02/compact.done) = No such file or directory: 1.756 sec on http probe failed: 1, retry 2024/11/19-15:14:50.482336 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-02 2024/11/19-15:14:50.482422 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00004/att-02/compact.done) = No such file or directory: 3.166 sec on http probe failed: 2, retry 2024/11/19-15:14:50.586614 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-01 2024/11/19-15:14:50.586683 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00008/att-01/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:14:50.604235 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-01 2024/11/19-15:14:50.604297 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00009/att-01/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:14:50.622616 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-01 2024/11/19-15:14:50.622671 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00006/att-01/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:14:50.626446 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-01 2024/11/19-15:14:50.626492 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00007/att-01/compact.done) = No such file or directory: 7.193 sec on http probe failed: 3, retry 2024/11/19-15:14:50.635093 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-01 2024/11/19-15:14:50.635145 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00005/att-01/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:14:50.667332 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-01 2024/11/19-15:14:50.667381 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00003/att-01/compact.done) = No such file or directory: 7.210 sec on http probe failed: 3, retry 2024/11/19-15:14:51.896845 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-02 2024/11/19-15:14:51.896917 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00004/att-02/compact.done) = No such file or directory: 4.581 sec on http probe failed: 3, retry 2024/11/19-15:14:52.594739 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-01 2024/11/19-15:14:52.594826 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00008/att-01/compact.done) = No such file or directory: 9.194 sec on http probe failed: 4, retry 2024/11/19-15:14:52.613779 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-01 2024/11/19-15:14:52.613837 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00009/att-01/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:14:52.630901 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-01 2024/11/19-15:14:52.630965 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00006/att-01/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:14:52.634529 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-01 2024/11/19-15:14:52.634574 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00007/att-01/compact.done) = No such file or directory: 9.203 sec on http probe failed: 4, retry 2024/11/19-15:14:52.639810 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-01 2024/11/19-15:14:52.639874 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00005/att-01/compact.done) = No such file or directory: 9.193 sec on http probe failed: 4, retry 2024/11/19-15:14:52.675641 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-01 2024/11/19-15:14:52.675720 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00003/att-01/compact.done) = No such file or directory: 9.216 sec on http probe failed: 4, retry 2024/11/19-15:14:53.311065 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-02 2024/11/19-15:14:53.311159 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00004/att-02/compact.done) = No such file or directory: 5.995 sec on http probe failed: 4, retry 2024/11/19-15:14:54.603238 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-01 2024/11/19-15:14:54.603328 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00008/att-01/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:14:54.603347 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00008/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.870 sec, real 11.507 (submit 296.012 ms wait 11.202 s), input 126.870 MiB, send shutdown 2024/11/19-15:14:54.608880 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-01 2024/11/19-15:14:54.609107 139851137550080 [dcompact_etcd.cc:1773] job-00008/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:14:54.618178 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-01 2024/11/19-15:14:54.618232 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00009/att-01/compact.done) = No such file or directory: 11.205 sec on http probe failed: 5, giveup 2024/11/19-15:14:54.618247 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00009/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.908 sec, real 11.507 (submit 297.016 ms wait 11.205 s), input 126.908 MiB, send shutdown 2024/11/19-15:14:54.626306 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-01 2024/11/19-15:14:54.626480 139851127064320 [dcompact_etcd.cc:1773] job-00009/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:14:54.638911 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-01 2024/11/19-15:14:54.638993 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00006/att-01/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:14:54.639009 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00006/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.925 sec, real 11.526 (submit 311.398 ms wait 11.207 s), input 126.925 MiB, send shutdown 2024/11/19-15:14:54.643055 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-01 2024/11/19-15:14:54.643104 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00007/att-01/compact.done) = No such file or directory: 11.211 sec on http probe failed: 5, giveup 2024/11/19-15:14:54.643117 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00007/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.998 sec, real 11.512 (submit 292.448 ms wait 11.211 s), input 126.998 MiB, send shutdown 2024/11/19-15:14:54.645773 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-01 2024/11/19-15:14:54.645822 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00005/att-01/compact.done) = No such file or directory: 11.198 sec on http probe failed: 5, giveup 2024/11/19-15:14:54.645835 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00005/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.886 sec, real 11.509 (submit 304.738 ms wait 11.198 s), input 126.886 MiB, send shutdown 2024/11/19-15:14:54.647517 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-01 2024/11/19-15:14:54.647693 139851158521600 [dcompact_etcd.cc:1773] job-00006/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:14:54.651050 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-01 2024/11/19-15:14:54.651228 139851148035840 [dcompact_etcd.cc:1773] job-00007/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:14:54.653812 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-01 2024/11/19-15:14:54.653977 139851169007360 [dcompact_etcd.cc:1773] job-00005/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:14:54.684027 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-01 2024/11/19-15:14:54.684078 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00003/att-01/compact.done) = No such file or directory: 11.224 sec on http probe failed: 5, giveup 2024/11/19-15:14:54.684092 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00003/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.822 sec, real 11.533 (submit 300.647 ms wait 11.224 s), input 127.822 MiB, send shutdown 2024/11/19-15:14:54.692802 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-01 2024/11/19-15:14:54.692973 139850978166528 [dcompact_etcd.cc:1773] job-00003/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:14:54.723092 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-02 2024/11/19-15:14:54.723140 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00004/att-02/compact.done) = No such file or directory: 7.409 sec on http probe failed: 5, giveup 2024/11/19-15:14:54.723154 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00004/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 70.230 sec, real 7.687 (submit 270.470 ms wait 7.409 s), input 70.230 MiB, send shutdown 2024/11/19-15:14:54.728367 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00004/att-02 2024/11/19-15:14:54.728539 139851179493120 [dcompact_etcd.cc:1773] job-00004/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:14:54.730341 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00004) = success 2024/11/19-15:14:54.730350 139851179493120 [dcompact_etcd.cc:1773] job-00004/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:14:54.730377 139851179493120 (Original Log Time 2024/11/19-15:14:31.586339) EVENT_LOG_v1 {"time_micros": 1732000471586330, "job": 4, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4243, 4149, 4154], "score": 4.89028, "input_data_size": 73354249, "oldest_snapshot_seqno": -1} 2024/11/19-15:14:54.730384 139851179493120 [compaction_job.cc:2437] [default] [JOB 4] Compacting 3@1 files to L2, score 4.89, subcompactions 1 : 1 2024/11/19-15:14:54.730391 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 6 Base level 1, inputs: [4243(26MB) 4149(31MB) 4154(11MB)] 2024/11/19-15:14:54.730669 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004322.sst) /storage/mytopling/..//tmp/Topling-82038-BUb2Qm 2024/11/19-15:14:54.869820 139851137550080 [dcompact_etcd.cc:1163] job-00008/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.261 sec, input = 126.870 MiB 2024/11/19-15:14:54.876269 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004322.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:14:54.906787 139851148035840 [dcompact_etcd.cc:1163] job-00007/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.255 sec, input = 126.998 MiB 2024/11/19-15:14:54.912515 139851127064320 [dcompact_etcd.cc:1163] job-00009/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.286 sec, input = 126.908 MiB 2024/11/19-15:14:54.913035 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004322.sst): index pass time = 0.04's, 201.853'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463343 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:14:54.918857 139851169007360 [dcompact_etcd.cc:1163] job-00005/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.265 sec, input = 126.886 MiB 2024/11/19-15:14:54.936374 139851158521600 [dcompact_etcd.cc:1163] job-00006/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 126.925 MiB 2024/11/19-15:14:54.979873 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004322.sst): first pass time = 0.25's, 269.309'MB/sec 2024/11/19-15:14:54.979950 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004322.sst): old prealloc_size = 73819750, real_size = 55604976 2024/11/19-15:14:54.980185 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004322.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463343 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.84 avg-zval = 117.87 2024/11/19-15:14:54.980246 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004322.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.104 sec, Key+Value = 0.067 GB 2024/11/19-15:14:54.990352 139850978166528 [dcompact_etcd.cc:1163] job-00003/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.297 sec, input = 127.822 MiB 2024/11/19-15:14:55.069438 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004322.sst): second pass time = 0.00's,94974207.018'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 620.167'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3835.021'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.587'MB/sec write SST data time = 0.04's, 1247.561'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 94974207.018'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463343 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.84 avg-zval = 117.87 usrkeys = 463343 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.84 avg-zval = 117.87 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6616 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5720 value = 0.9913 dict = 0.00 all = 1.2093 } Zip/UnZip{ index = 0.0946 value = 1.0088 dict = 0.00 all = 0.8269 } ---------------------------- total value len = 0.054135 GB avg = 0.117 KB (by entry num) total key len = 0.012974 GB avg = 0.028 KB total ukey len = 0.009267 GB avg = 0.020 KB total ukey num = 0.000463343 Billion total entry num = 0.000463343 Billion write speed all = 198.087498406 MB/sec (with seq num) write speed all = 187.146183256 MB/sec (without seq num) 2024/11/19-15:14:55.073264 139851179493120 [compaction_job.cc:1948] [default] [JOB 4] Generated table #4322: 463343 keys, 55494385 bytes (need compaction), temperature: kUnknown 2024/11/19-15:14:55.073325 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000495073299, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 4322, "file_size": 55494385, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54612384, "index_size": 876544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12973604, "raw_average_key_size": 28, "raw_value_size": 54135298, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463343, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000494, "oldest_key_time": 0, "file_creation_time": 1732000494, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4322, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:14:55.087711 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004323.sst) /storage/mytopling/..//tmp/Topling-82038-wbukwL 2024/11/19-15:14:55.101899 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004323.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.007 GB 2024/11/19-15:14:55.106836 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004323.sst): index pass time = 0.00's, 147.924'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 45242 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 7 raw-key = 0.0007 GB zip-key = 0.0001 GB avg-key = 16.00 avg-zkey = 1.93 2024/11/19-15:14:55.112344 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004323.sst): first pass time = 0.02's, 265.388'MB/sec 2024/11/19-15:14:55.112381 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004323.sst): old prealloc_size = 73819750, real_size = 5419920 2024/11/19-15:14:55.112430 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004323.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 45242 prefix = 4 raw-key = 0.0007 GB zip-key = 0.0001 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.0053 GB zip-val = 0.0053 GB avg-val = 116.39 avg-zval = 117.61 2024/11/19-15:14:55.112463 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004323.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.011 sec, Key+Value = 0.007 GB 2024/11/19-15:14:55.121011 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004323.sst): second pass time = 0.00's,26595191.919'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.01's, 627.973'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2005.961'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 2.631'MB/sec write SST data time = 0.00's, 1266.105'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.005'GB zip my value throughput = 26595191.919'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 45242 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.39 avg-zval = 117.61 usrkeys = 45242 avg-key = 20.00 avg-zkey = 1.93 avg-val = 116.39 avg-zval = 117.61 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 648 UnZipSize{ index = 0.0009 GB value = 0.0053 GB dict = 0.00 MB all = 0.0065 GB } __ZipSize{ index = 0.0001 GB value = 0.0053 GB dict = 0.00 MB all = 0.0054 GB } UnZip/Zip{ index = 10.3424 value = 0.9896 dict = 0.00 all = 1.2075 } Zip/UnZip{ index = 0.0967 value = 1.0105 dict = 0.00 all = 0.8281 } ---------------------------- total value len = 0.059401 GB avg = 0.117 KB (by entry num) total key len = 0.014240 GB avg = 0.028 KB total ukey len = 0.010172 GB avg = 0.020 KB total ukey num = 0.000508585 Billion total entry num = 0.000508585 Billion write speed all = 188.645372160 MB/sec (with seq num) write speed all = 178.222752010 MB/sec (without seq num) 2024/11/19-15:14:55.121551 139851179493120 [compaction_job.cc:1948] [default] [JOB 4] Generated table #4323: 45242 keys, 5409904 bytes (need compaction), temperature: kUnknown 2024/11/19-15:14:55.121595 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000495121573, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 4323, "file_size": 5409904, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 5320960, "index_size": 87488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1266776, "raw_average_key_size": 28, "raw_value_size": 5265848, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 45242, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000495, "oldest_key_time": 0, "file_creation_time": 1732000495, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4323, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:14:55.123029 139851179493120 [compaction_job.cc:703] job-00004: subcompact[0], size: 0.060904 G, files: /storage/mytopling/.rocksdb [004322,004323] 2024/11/19-15:14:55.124154 139851179493120 (Original Log Time 2024/11/19-15:14:54.730418) EVENT_LOG_v1 {"time_micros": 1732000494730404, "job": 4, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4243, 4149, 4154], "score": 4.89028, "input_data_size": 73354249, "oldest_snapshot_seqno": -1} 2024/11/19-15:14:55.124163 139851179493120 (Original Log Time 2024/11/19-15:14:55.123708) [compaction_job.cc:2022] [default] [JOB 4] Compacted 3@1 files to L2 => 60904289 bytes 2024/11/19-15:14:55.124167 139851179493120 (Original Log Time 2024/11/19-15:14:55.124046) [compaction_job.cc:1224] [default] compacted to: files[3 81 62 75 0 0 0] max score 3.50 (122 files need compaction), MB/sec: 186.8 rd, 155.1 wr, level 2, files in(3, 0) out(2 +0 blob) MB in(70.0, 0.0 +0.0 blob) out(58.1 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 508585, records dropped: 0 output_compression: Snappy 2024/11/19-15:14:55.124173 139851179493120 (Original Log Time 2024/11/19-15:14:55.124076) EVENT_LOG_v1 {"time_micros": 1732000495124060, "cf": "default", "job": 4, "event": "compaction_finished", "compaction_time_micros": 392621, "compaction_time_cpu_micros": 389671, "output_level": 2, "num_output_files": 2, "total_output_size": 60904289, "num_input_records": 508585, "num_output_records": 508585, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 81, 62, 75, 0, 0, 0]} 2024/11/19-15:14:55.124279 139851179493120 [compaction_job.cc:2437] [default] [JOB 12] Compacting 4@1 files to L2, score 3.50, subcompactions 1 : 1 2024/11/19-15:14:55.124288 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 10 Base level 1, inputs: [4193(31MB) 4194(31MB) 4196(31MB) 4197(31MB)] 2024/11/19-15:14:55.434258 139851179493120 [dcompact_etcd.cc:1163] job-00012/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.309 sec, input = 126.976 MiB 2024/11/19-15:14:58.047058 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-02 2024/11/19-15:14:58.047169 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00008/att-02/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:14:58.087802 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-02 2024/11/19-15:14:58.087876 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00007/att-02/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:14:58.090874 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-02 2024/11/19-15:14:58.090943 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00009/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:14:58.100448 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-02 2024/11/19-15:14:58.100528 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00005/att-02/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:14:58.115029 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-02 2024/11/19-15:14:58.115088 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00006/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:14:58.195242 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-02 2024/11/19-15:14:58.195338 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00003/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:14:58.614870 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-00 2024/11/19-15:14:58.614988 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00012/att-00/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:15:00.057451 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-02 2024/11/19-15:15:00.057554 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00008/att-02/compact.done) = No such file or directory: 5.177 sec on http probe failed: 2, retry 2024/11/19-15:15:00.094236 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-02 2024/11/19-15:15:00.094294 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00007/att-02/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:15:00.099881 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-02 2024/11/19-15:15:00.099942 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00009/att-02/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:15:00.106039 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-02 2024/11/19-15:15:00.106091 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00005/att-02/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:15:00.119363 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-02 2024/11/19-15:15:00.119413 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00006/att-02/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:15:00.201843 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-02 2024/11/19-15:15:00.201940 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00003/att-02/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:15:00.623534 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-00 2024/11/19-15:15:00.623608 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00012/att-00/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:15:02.073137 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-02 2024/11/19-15:15:02.073235 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00008/att-02/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:15:02.103176 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-02 2024/11/19-15:15:02.103246 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00007/att-02/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:15:02.108408 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-02 2024/11/19-15:15:02.108464 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00009/att-02/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:15:02.116402 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-02 2024/11/19-15:15:02.116453 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00005/att-02/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:15:02.127443 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-02 2024/11/19-15:15:02.127493 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00006/att-02/compact.done) = No such file or directory: 7.183 sec on http probe failed: 3, retry 2024/11/19-15:15:02.210436 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-02 2024/11/19-15:15:02.210500 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00003/att-02/compact.done) = No such file or directory: 7.212 sec on http probe failed: 3, retry 2024/11/19-15:15:02.628683 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-00 2024/11/19-15:15:02.628765 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00012/att-00/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:15:04.081903 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-02 2024/11/19-15:15:04.081990 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00008/att-02/compact.done) = No such file or directory: 9.203 sec on http probe failed: 4, retry 2024/11/19-15:15:04.107859 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-02 2024/11/19-15:15:04.107915 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00007/att-02/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:15:04.115718 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-02 2024/11/19-15:15:04.115774 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00009/att-02/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:15:04.125371 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-02 2024/11/19-15:15:04.125421 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00005/att-02/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:15:04.135642 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-02 2024/11/19-15:15:04.135693 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00006/att-02/compact.done) = No such file or directory: 9.191 sec on http probe failed: 4, retry 2024/11/19-15:15:04.215527 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-02 2024/11/19-15:15:04.215589 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00003/att-02/compact.done) = No such file or directory: 9.220 sec on http probe failed: 4, retry 2024/11/19-15:15:04.638603 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-00 2024/11/19-15:15:04.638682 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00012/att-00/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:15:05.602376 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4324. Immutable memtables: 0. 2024/11/19-15:15:05.608746 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 13] Syncing log #4320 2024/11/19-15:15:05.608867 139850967680768 (Original Log Time 2024/11/19-15:15:05.602499) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:15:05.608878 139850967680768 [flush_job.cc:879] [default] [JOB 13] Flushing memtable with next log file: 4324 2024/11/19-15:15:05.608900 139850967680768 EVENT_LOG_v1 {"time_micros": 1732000505608894, "job": 13, "event": "flush_started", "num_memtables": 1, "num_entries": 1227615, "num_deletes": 0, "total_data_size": 487800470, "memory_usage": 543423880, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:15:05.608904 139850967680768 [flush_job.cc:910] [default] [JOB 13] Level-0 flush table #4325: started 2024/11/19-15:15:05.609421 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000000.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004325.sst in ConvertToSST, mem_size = 518.249 M, time = 504.550 us 2024/11/19-15:15:06.087568 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-02 2024/11/19-15:15:06.087735 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00008/att-02/compact.done) = No such file or directory: 11.212 sec on http probe failed: 5, giveup 2024/11/19-15:15:06.087785 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00008/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.870 sec, real 11.479 (submit 260.621 ms wait 11.212 s), input 126.870 MiB, send shutdown 2024/11/19-15:15:06.093006 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00008/att-02 2024/11/19-15:15:06.114134 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-02 2024/11/19-15:15:06.114213 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00007/att-02/compact.done) = No such file or directory: 11.201 sec on http probe failed: 5, giveup 2024/11/19-15:15:06.114238 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00007/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.998 sec, real 11.463 (submit 255.489 ms wait 11.201 s), input 126.998 MiB, send shutdown 2024/11/19-15:15:06.119341 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00007/att-02 2024/11/19-15:15:06.121415 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-02 2024/11/19-15:15:06.121478 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00009/att-02/compact.done) = No such file or directory: 11.203 sec on http probe failed: 5, giveup 2024/11/19-15:15:06.121492 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00009/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.908 sec, real 11.495 (submit 285.963 ms wait 11.203 s), input 126.908 MiB, send shutdown 2024/11/19-15:15:06.126433 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00009/att-02 2024/11/19-15:15:06.135068 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-02 2024/11/19-15:15:06.135175 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00005/att-02/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:15:06.135208 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00005/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.886 sec, real 11.481 (submit 264.818 ms wait 11.207 s), input 126.886 MiB, send shutdown 2024/11/19-15:15:06.143630 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00005/att-02 2024/11/19-15:15:06.143941 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-02 2024/11/19-15:15:06.143997 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00006/att-02/compact.done) = No such file or directory: 11.199 sec on http probe failed: 5, giveup 2024/11/19-15:15:06.144011 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00006/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.925 sec, real 11.496 (submit 288.605 ms wait 11.199 s), input 126.925 MiB, send shutdown 2024/11/19-15:15:06.148682 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00006/att-02 2024/11/19-15:15:06.169231 139851137550080 [dcompact_etcd.cc:1773] job-00008/att-02: CleanFiles 0.076 sec : 0.000 2024/11/19-15:15:06.223676 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-02 2024/11/19-15:15:06.223748 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00003/att-02/compact.done) = No such file or directory: 11.225 sec on http probe failed: 5, giveup 2024/11/19-15:15:06.223764 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00003/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.822 sec, real 11.531 (submit 297.310 ms wait 11.225 s), input 127.822 MiB, send shutdown 2024/11/19-15:15:06.231647 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00003/att-02 2024/11/19-15:15:06.271126 139851148035840 [dcompact_etcd.cc:1773] job-00007/att-02: CleanFiles 0.152 sec : 0.000 2024/11/19-15:15:06.292067 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004325.sst): fsize = 518.251 M, time(ms): open: 0.037, seek: 0.029, finish: 0.036, meta: 0.015, Flush: 0.012, sync: 682.493, close: 0.008, all: 682.630 2024/11/19-15:15:06.293202 139850967680768 [flush_job.cc:1083] [default] [JOB 13] Flush lasted 684336 microseconds, and 399169 cpu microseconds. 2024/11/19-15:15:06.315562 139850967680768 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:15:06.315610 139850967680768 (Original Log Time 2024/11/19-15:15:06.292110) [flush_job.cc:1029] [default] [JOB 13] Level-0 flush table #4325: 543425306 bytes OK (needs compaction) 2024/11/19-15:15:06.315616 139850967680768 (Original Log Time 2024/11/19-15:15:06.293219) [memtable_list.cc:560] [default] Level-0 commit table #4325 started 2024/11/19-15:15:06.315619 139850967680768 (Original Log Time 2024/11/19-15:15:06.315528) [memtable_list.cc:759] [default] Level-0 commit table #4325: memtable #1 done 2024/11/19-15:15:06.315622 139850967680768 (Original Log Time 2024/11/19-15:15:06.315555) EVENT_LOG_v1 {"time_micros": 1732000506315547, "job": 13, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [4, 81, 62, 75, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:15:06.315625 139850967680768 (Original Log Time 2024/11/19-15:15:06.315584) [db_impl_compaction_flush.cc:310] [default] Level summary: files[4 81 62 75 0 0 0] max score 3.25 (119 files need compaction) 2024/11/19-15:15:06.347827 139851127064320 [dcompact_etcd.cc:1773] job-00009/att-02: CleanFiles 0.221 sec : 0.000 2024/11/19-15:15:06.347916 139851169007360 [dcompact_etcd.cc:1773] job-00005/att-02: CleanFiles 0.204 sec : 0.000 2024/11/19-15:15:06.347972 139851158521600 [dcompact_etcd.cc:1773] job-00006/att-02: CleanFiles 0.199 sec : 0.000 2024/11/19-15:15:06.348106 139850978166528 [dcompact_etcd.cc:1773] job-00003/att-02: CleanFiles 0.116 sec : 0.000 2024/11/19-15:15:06.348366 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00008) = success 2024/11/19-15:15:06.348374 139851137550080 [dcompact_etcd.cc:1773] job-00008/att-03: CleanFiles 0.179 sec : 0.000 2024/11/19-15:15:06.348393 139851137550080 (Original Log Time 2024/11/19-15:14:31.586679) EVENT_LOG_v1 {"time_micros": 1732000471586668, "job": 8, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4182, 4183, 4185, 4186], "score": 4.00374, "input_data_size": 134100928, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:06.348401 139851137550080 [compaction_job.cc:2437] [default] [JOB 8] Compacting 4@1 files to L2, score 4.00, subcompactions 1 : 1 2024/11/19-15:15:06.348409 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 6 Base level 1, inputs: [4182(31MB) 4183(31MB) 4185(31MB) 4186(31MB)] 2024/11/19-15:15:06.348790 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00007) = success 2024/11/19-15:15:06.348799 139851148035840 [dcompact_etcd.cc:1773] job-00007/att-03: CleanFiles 0.078 sec : 0.000 2024/11/19-15:15:06.348819 139851148035840 (Original Log Time 2024/11/19-15:14:31.586692) EVENT_LOG_v1 {"time_micros": 1732000471586682, "job": 7, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4176, 4178, 4179, 4181], "score": 4.25352, "input_data_size": 134099008, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:06.348827 139851148035840 [compaction_job.cc:2437] [default] [JOB 7] Compacting 4@1 files to L2, score 4.25, subcompactions 1 : 1 2024/11/19-15:15:06.348838 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 6 Base level 1, inputs: [4176(31MB) 4178(31MB) 4179(31MB) 4181(31MB)] 2024/11/19-15:15:06.348906 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004326.sst) /storage/mytopling/..//tmp/Topling-82038-dILvTA 2024/11/19-15:15:06.349102 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004327.sst) /storage/mytopling/..//tmp/Topling-82038-Jc1QV1 2024/11/19-15:15:06.368517 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00009) = success 2024/11/19-15:15:06.368538 139851127064320 [dcompact_etcd.cc:1773] job-00009/att-03: CleanFiles 0.021 sec : 0.000 2024/11/19-15:15:06.368557 139851127064320 (Original Log Time 2024/11/19-15:14:31.586711) EVENT_LOG_v1 {"time_micros": 1732000471586702, "job": 9, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4187, 4188, 4190, 4191], "score": 3.75396, "input_data_size": 134099328, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:06.368564 139851127064320 [compaction_job.cc:2437] [default] [JOB 9] Compacting 4@1 files to L2, score 3.75, subcompactions 1 : 1 2024/11/19-15:15:06.368572 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 6 Base level 1, inputs: [4187(31MB) 4188(31MB) 4190(31MB) 4191(31MB)] 2024/11/19-15:15:06.378572 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00005) = success 2024/11/19-15:15:06.378591 139851169007360 [dcompact_etcd.cc:1773] job-00005/att-03: CleanFiles 0.031 sec : 0.000 2024/11/19-15:15:06.378608 139851169007360 (Original Log Time 2024/11/19-15:14:31.586380) EVENT_LOG_v1 {"time_micros": 1732000471586371, "job": 5, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4157, 4160, 4162, 4164], "score": 4.75308, "input_data_size": 134099648, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:06.378618 139851169007360 [compaction_job.cc:2437] [default] [JOB 5] Compacting 4@1 files to L2, score 4.75, subcompactions 1 : 1 2024/11/19-15:15:06.378628 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 6 Base level 1, inputs: [4157(31MB) 4160(31MB) 4162(31MB) 4164(31MB)] 2024/11/19-15:15:06.389086 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00006) = success 2024/11/19-15:15:06.389100 139851158521600 [dcompact_etcd.cc:1773] job-00006/att-03: CleanFiles 0.041 sec : 0.000 2024/11/19-15:15:06.389116 139851158521600 (Original Log Time 2024/11/19-15:14:31.586447) EVENT_LOG_v1 {"time_micros": 1732000471586436, "job": 6, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4168, 4170, 4173, 4175], "score": 4.5033, "input_data_size": 134098624, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:06.389124 139851158521600 [compaction_job.cc:2437] [default] [JOB 6] Compacting 4@1 files to L2, score 4.50, subcompactions 1 : 1 2024/11/19-15:15:06.389132 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 6 Base level 1, inputs: [4168(31MB) 4170(31MB) 4173(31MB) 4175(31MB)] 2024/11/19-15:15:06.403796 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004326.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:15:06.406701 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004327.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:15:06.406816 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004327.sst): index pass time = 0.00's,7531.116'MB/sec index type = UintIndex_AllOne usrkeys = 64713 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:15:06.407722 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004326.sst): index pass time = 0.00's, 133.561'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64713 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:15:06.407823 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00003) = success 2024/11/19-15:15:06.407843 139850978166528 [dcompact_etcd.cc:1773] job-00003/att-03: CleanFiles 0.060 sec : 0.000 2024/11/19-15:15:06.407871 139850978166528 (Original Log Time 2024/11/19-15:14:31.586329) EVENT_LOG_v1 {"time_micros": 1732000471586310, "job": 3, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4227, 4229, 4235, 4239], "score": 5.13994, "input_data_size": 133300108, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:06.407884 139850978166528 [compaction_job.cc:2437] [default] [JOB 3] Compacting 4@1 files to L2, score 5.14, subcompactions 1 : 1 2024/11/19-15:15:06.407897 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 6 Base level 1, inputs: [4227(31MB) 4229(31MB) 4235(31MB) 4239(31MB)] 2024/11/19-15:15:06.407954 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004328.sst) /storage/mytopling/..//tmp/Topling-82038-p7fwbY 2024/11/19-15:15:06.407960 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004329.sst) /storage/mytopling/..//tmp/Topling-82038-8nmaOt 2024/11/19-15:15:06.408064 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004330.sst) /storage/mytopling/..//tmp/Topling-82038-a2PvJ8 2024/11/19-15:15:06.408225 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004331.sst) /storage/mytopling/..//tmp/Topling-82038-exCSpn 2024/11/19-15:15:06.492481 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004328.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:15:06.498280 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004326.sst): first pass time = 0.15's, 449.348'MB/sec 2024/11/19-15:15:06.498328 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004326.sst): old prealloc_size = 73819750, real_size = 66152160 2024/11/19-15:15:06.498384 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004326.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64713 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.03 avg-zval =1020.28 2024/11/19-15:15:06.498429 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004326.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.095 sec, Key+Value = 0.067 GB 2024/11/19-15:15:06.498527 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004328.sst): index pass time = 0.01's, 86.219'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64717 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:15:06.500790 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004327.sst): first pass time = 0.15's, 442.452'MB/sec 2024/11/19-15:15:06.500840 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004327.sst): old prealloc_size = 73819750, real_size = 66041832 2024/11/19-15:15:06.500870 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004327.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64713 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.03 avg-zval =1020.28 2024/11/19-15:15:06.501583 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004330.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:15:06.501664 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004330.sst): index pass time = 0.00's,9051.365'MB/sec index type = UintIndex_AllOne usrkeys = 64715 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:15:06.510958 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004329.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:15:06.523127 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004329.sst): index pass time = 0.01's, 42.728'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64713 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:15:06.576320 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004327.sst): second pass time = 0.00's,288661399.123'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.08's, 875.160'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 7.863'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.303'MB/sec write SST data time = 0.02's, 2986.724'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 288661399.123'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64713 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.03 avg-zval = 1020.28 usrkeys = 64713 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.03 avg-zval = 1020.28 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.5625 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 0.125216 GB avg = 0.218 KB (by entry num) total key len = 0.015535 GB avg = 0.027 KB total ukey len = 0.010948 GB avg = 0.019 KB total ukey num = 0.000573298 Billion total entry num = 0.000573298 Billion write speed all = 11.882049427 MB/sec (with seq num) write speed all = 11.494870636 MB/sec (without seq num) 2024/11/19-15:15:06.592537 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004328.sst): first pass time = 0.18's, 363.611'MB/sec 2024/11/19-15:15:06.592585 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004328.sst): old prealloc_size = 73819750, real_size = 66152032 2024/11/19-15:15:06.592644 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004328.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64717 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.97 avg-zval =1020.22 2024/11/19-15:15:06.592689 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004328.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.100 sec, Key+Value = 0.067 GB 2024/11/19-15:15:06.600204 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004330.sst): first pass time = 0.19's, 349.307'MB/sec 2024/11/19-15:15:06.600307 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004330.sst): old prealloc_size = 73819750, real_size = 66041784 2024/11/19-15:15:06.600334 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004330.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64715 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.99 avg-zval =1020.25 2024/11/19-15:15:06.606287 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004326.sst): second pass time = 0.00's,142456878.788'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.11's, 612.801'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2092.107'MB/sec (index lex order gen) rebuild zvType time = 0.08's, 0.194'MB/sec write SST data time = 0.02's, 2700.847'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 142456878.788'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64713 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 1020.28 usrkeys = 64713 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 1020.28 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0463 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 0.191031 GB avg = 0.299 KB (by entry num) total key len = 0.016829 GB avg = 0.026 KB total ukey len = 0.011725 GB avg = 0.018 KB total ukey num = 0.000638011 Billion total entry num = 0.000638011 Billion write speed all = 17.503074490 MB/sec (with seq num) write speed all = 17.073279120 MB/sec (without seq num) 2024/11/19-15:15:06.611814 139851148035840 [compaction_job.cc:1948] [default] [JOB 7] Generated table #4327: 64713 keys, 66026611 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:06.611881 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000506611848, "cf_name": "default", "job": 7, "event": "table_file_creation", "file_number": 4327, "file_size": 66026611, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025328, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294260, "raw_average_key_size": 20, "raw_value_size": 65814799, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64713, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000506, "oldest_key_time": 0, "file_creation_time": 1732000506, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4327, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:06.614548 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004329.sst): first pass time = 0.21's, 324.865'MB/sec 2024/11/19-15:15:06.614626 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004329.sst): old prealloc_size = 73819750, real_size = 66151824 2024/11/19-15:15:06.614695 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004329.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64713 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.03 avg-zval =1020.28 2024/11/19-15:15:06.614750 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004329.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.104 sec, Key+Value = 0.067 GB 2024/11/19-15:15:06.635426 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004332.sst) /storage/mytopling/..//tmp/Topling-82038-1189Cv 2024/11/19-15:15:06.645300 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-00 2024/11/19-15:15:06.645390 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00012/att-00/compact.done) = No such file or directory: 11.205 sec on http probe failed: 5, giveup 2024/11/19-15:15:06.645413 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00012/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.976 sec, real 11.521 (submit 309.481 ms wait 11.205 s), input 126.976 MiB, send shutdown 2024/11/19-15:15:06.654952 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-00 2024/11/19-15:15:06.658749 139851137550080 [compaction_job.cc:1948] [default] [JOB 8] Generated table #4326: 64713 keys, 66137389 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:06.658817 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000506658784, "cf_name": "default", "job": 8, "event": "table_file_creation", "file_number": 4326, "file_size": 66137389, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025616, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294260, "raw_average_key_size": 20, "raw_value_size": 65815078, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64713, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000506, "oldest_key_time": 0, "file_creation_time": 1732000506, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4326, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:06.659937 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004328.sst): second pass time = 0.00's,198237774.096'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 982.704'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2082.933'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.328'MB/sec write SST data time = 0.02's, 3676.738'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 198237774.096'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64717 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.97 avg-zval = 1020.22 usrkeys = 64717 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.97 avg-zval = 1020.22 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0467 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 0.256846 GB avg = 0.365 KB (by entry num) total key len = 0.018123 GB avg = 0.026 KB total ukey len = 0.012501 GB avg = 0.018 KB total ukey num = 0.000702728 Billion total entry num = 0.000702728 Billion write speed all = 23.049986365 MB/sec (with seq num) write speed all = 22.578722803 MB/sec (without seq num) 2024/11/19-15:15:06.669179 139851179493120 [dcompact_etcd.cc:1773] job-00012/att-00: CleanFiles 0.014 sec : 0.000 2024/11/19-15:15:06.678476 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004333.sst) /storage/mytopling/..//tmp/Topling-82038-cptbmX 2024/11/19-15:15:06.679437 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004331.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:15:06.686427 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004330.sst): second pass time = 0.00's,262210135.458'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 766.482'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 7.653'MB/sec (index lex order gen) rebuild zvType time = 0.06's, 0.271'MB/sec write SST data time = 0.03's, 2498.770'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 262210135.458'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64715 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.99 avg-zval = 1020.25 usrkeys = 64715 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.99 avg-zval = 1020.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.6875 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 0.322661 GB avg = 0.420 KB (by entry num) total key len = 0.019418 GB avg = 0.025 KB total ukey len = 0.013278 GB avg = 0.017 KB total ukey num = 0.000767443 Billion total entry num = 0.000767443 Billion write speed all = 28.611986581 MB/sec (with seq num) write speed all = 28.098465035 MB/sec (without seq num) 2024/11/19-15:15:06.698688 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004329.sst): second pass time = 0.00's,120319458.867'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.08's, 786.817'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1657.612'MB/sec (index lex order gen) rebuild zvType time = 0.06's, 0.257'MB/sec write SST data time = 0.02's, 3136.280'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 120319458.867'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64713 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 1020.28 usrkeys = 64713 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 1020.28 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8063 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0463 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 0.388475 GB avg = 0.467 KB (by entry num) total key len = 0.020712 GB avg = 0.025 KB total ukey len = 0.014055 GB avg = 0.017 KB total ukey num = 0.000832156 Billion total entry num = 0.000832156 Billion write speed all = 34.189992906 MB/sec (with seq num) write speed all = 33.633740821 MB/sec (without seq num) 2024/11/19-15:15:06.698926 139851127064320 [compaction_job.cc:1948] [default] [JOB 9] Generated table #4328: 64717 keys, 66137261 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:06.699002 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000506698962, "cf_name": "default", "job": 9, "event": "table_file_creation", "file_number": 4328, "file_size": 66137261, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025488, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294340, "raw_average_key_size": 20, "raw_value_size": 65814941, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64717, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000506, "oldest_key_time": 0, "file_creation_time": 1732000506, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4328, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:06.708377 139851158521600 [compaction_job.cc:1948] [default] [JOB 6] Generated table #4330: 64715 keys, 66026547 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:06.708428 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000506708405, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 4330, "file_size": 66026547, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025280, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294300, "raw_average_key_size": 20, "raw_value_size": 65814744, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64715, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000506, "oldest_key_time": 0, "file_creation_time": 1732000506, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4330, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:06.716784 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004332.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.006 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:15:06.722614 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004334.sst) /storage/mytopling/..//tmp/Topling-82038-hbzizW 2024/11/19-15:15:06.725822 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004332.sst): index pass time = 0.01's, 56.576'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63699 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:15:06.728963 139851169007360 [compaction_job.cc:1948] [default] [JOB 5] Generated table #4329: 64713 keys, 66137005 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:06.729029 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000506728998, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 4329, "file_size": 66137005, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025280, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294260, "raw_average_key_size": 20, "raw_value_size": 65814744, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64713, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000506, "oldest_key_time": 0, "file_creation_time": 1732000506, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4329, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:06.734009 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004331.sst): index pass time = 0.05's, 135.930'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463343 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:15:06.738764 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004335.sst) /storage/mytopling/..//tmp/Topling-82038-4kyJ4o 2024/11/19-15:15:06.756063 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004336.sst) /storage/mytopling/..//tmp/Topling-82038-ubUeHk 2024/11/19-15:15:06.764271 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004333.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:15:06.764377 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004333.sst): index pass time = 0.00's,7310.563'MB/sec index type = UintIndex_AllOne usrkeys = 63569 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:15:06.796198 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004332.sst): first pass time = 0.16's, 410.960'MB/sec 2024/11/19-15:15:06.796256 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004332.sst): old prealloc_size = 73819750, real_size = 65115840 2024/11/19-15:15:06.796317 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004332.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63699 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0648 GB zip-val = 0.0650 GB avg-val =1017.03 avg-zval =1020.29 2024/11/19-15:15:06.796359 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004332.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.080 sec, Key+Value = 0.066 GB 2024/11/19-15:15:06.804671 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004334.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:15:06.809667 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004334.sst): index pass time = 0.00's, 102.674'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63606 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:15:06.816942 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004335.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:15:06.822075 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004335.sst): index pass time = 0.01's, 99.805'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63630 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:15:06.834682 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004331.sst): first pass time = 0.43's, 157.371'MB/sec 2024/11/19-15:15:06.834753 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004331.sst): old prealloc_size = 73819750, real_size = 55605104 2024/11/19-15:15:06.835069 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004331.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463343 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.84 avg-zval = 117.87 2024/11/19-15:15:06.835152 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004331.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.156 sec, Key+Value = 0.067 GB 2024/11/19-15:15:06.841883 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004336.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.001 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:15:06.852338 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004336.sst): index pass time = 0.01's, 48.808'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63589 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:15:06.858664 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004333.sst): first pass time = 0.18's, 365.894'MB/sec 2024/11/19-15:15:06.858725 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004333.sst): old prealloc_size = 73819750, real_size = 64875144 2024/11/19-15:15:06.858763 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004333.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63569 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1017.04 avg-zval =1020.29 2024/11/19-15:15:06.868613 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004332.sst): second pass time = 0.00's,97713473.605'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 899.849'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1942.380'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.301'MB/sec write SST data time = 0.02's, 3346.121'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 97713473.605'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63699 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 1020.29 usrkeys = 63699 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 1020.29 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7938 UnZipSize{ index = 0.0008 GB value = 0.0648 GB dict = 0.00 MB all = 0.0661 GB } __ZipSize{ index = 0.0001 GB value = 0.0650 GB dict = 0.00 MB all = 0.0651 GB } UnZip/Zip{ index = 7.0463 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 0.453259 GB avg = 0.506 KB (by entry num) total key len = 0.021986 GB avg = 0.025 KB total ukey len = 0.014819 GB avg = 0.017 KB total ukey num = 0.000895855 Billion total entry num = 0.000895855 Billion write speed all = 39.153604791 MB/sec (with seq num) write speed all = 38.563156781 MB/sec (without seq num) 2024/11/19-15:15:06.873038 139851148035840 [compaction_job.cc:1948] [default] [JOB 7] Generated table #4332: 63699 keys, 65101293 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:06.873093 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000506873068, "cf_name": "default", "job": 7, "event": "table_file_creation", "file_number": 4332, "file_size": 65101293, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64991280, "index_size": 108480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1273980, "raw_average_key_size": 20, "raw_value_size": 64784033, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63699, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000506, "oldest_key_time": 0, "file_creation_time": 1732000506, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4332, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:06.888683 139851148035840 [compaction_job.cc:703] job-00007: subcompact[0], size: 0.131128 G, files: /storage/mytopling/.rocksdb [004327,004332] 2024/11/19-15:15:06.889512 139851148035840 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:15:06.889592 139851148035840 (Original Log Time 2024/11/19-15:15:06.348872) EVENT_LOG_v1 {"time_micros": 1732000506348850, "job": 7, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4176, 4178, 4179, 4181], "score": 4.25352, "input_data_size": 134099008, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:06.889598 139851148035840 (Original Log Time 2024/11/19-15:15:06.889134) [compaction_job.cc:2022] [default] [JOB 7] Compacted 4@1 files to L2 => 131127904 bytes 2024/11/19-15:15:06.889601 139851148035840 (Original Log Time 2024/11/19-15:15:06.889475) [compaction_job.cc:1224] [default] compacted to: files[4 77 64 75 0 0 0] max score 3.25 (121 files need compaction), MB/sec: 248.4 rd, 242.9 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.1 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128412, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:06.889609 139851148035840 (Original Log Time 2024/11/19-15:15:06.889503) EVENT_LOG_v1 {"time_micros": 1732000506889490, "cf": "default", "job": 7, "event": "compaction_finished", "compaction_time_micros": 539828, "compaction_time_cpu_micros": 499388, "output_level": 2, "num_output_files": 2, "total_output_size": 131127904, "num_input_records": 128412, "num_output_records": 128412, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 77, 64, 75, 0, 0, 0]} 2024/11/19-15:15:06.889750 139851148035840 [compaction_job.cc:2437] [__system__] [JOB 14] Compacting 1@0 + 1@1 files to L1, score 0.25, subcompactions 3 : 1 2024/11/19-15:15:06.889759 139851148035840 [compaction_job.cc:2445] [__system__]: Compaction start summary: Base version 7 Base level 0, inputs: [4317(3417B)], [4313(5672B)] 2024/11/19-15:15:06.890177 139851148035840 [compaction_job.cc:1948] [__system__] [JOB 14] Generated table #4337: 34 keys, 5672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:06.890214 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000506890194, "cf_name": "__system__", "job": 14, "event": "table_file_creation", "file_number": 4337, "file_size": 5672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 1216, "index_size": 3648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 803, "raw_average_key_size": 23, "raw_value_size": 1193, "raw_average_value_size": 35, "num_data_blocks": 1, "num_entries": 34, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "__system__", "column_family_id": 1, "comparator": "leveldb.BytewiseComparator", "merge_operator": "Rdb_system_merge_op", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000506, "oldest_key_time": 0, "file_creation_time": 1732000506, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4337, "seqno_to_time_mapping": "N/A"}} 2024/11/19-15:15:06.890251 139851148035840 [compaction_job.cc:703] job-00014: subcompact[0], size: 0.000006 G, files: /storage/mytopling/.rocksdb [004337] 2024/11/19-15:15:06.890495 139851148035840 (Original Log Time 2024/11/19-15:15:06.889778) EVENT_LOG_v1 {"time_micros": 1732000506889765, "job": 14, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L0": [4317], "files_L1": [4313], "score": 0.25, "input_data_size": 9089, "oldest_snapshot_seqno": 106912315} 2024/11/19-15:15:06.890503 139851148035840 (Original Log Time 2024/11/19-15:15:06.890395) [compaction_job.cc:2022] [__system__] [JOB 14] Compacted 1@0 + 1@1 files to L1 => 5672 bytes 2024/11/19-15:15:06.890506 139851148035840 (Original Log Time 2024/11/19-15:15:06.890434) [compaction_job.cc:1224] [__system__] compacted to: files[0 1 0 0 0 0 0] max score 0.00, MB/sec: 19.0 rd, 11.8 wr, level 1, files in(1, 1) out(1 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(4.3) write-amplify(1.7) OK, records in: 39, records dropped: 5 output_compression: Snappy 2024/11/19-15:15:06.890512 139851148035840 (Original Log Time 2024/11/19-15:15:06.890449) EVENT_LOG_v1 {"time_micros": 1732000506890442, "cf": "__system__", "job": 14, "event": "compaction_finished", "compaction_time_micros": 479, "compaction_time_cpu_micros": 372, "output_level": 1, "num_output_files": 1, "total_output_size": 5672, "num_input_records": 39, "num_output_records": 34, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]} 2024/11/19-15:15:06.890751 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004313.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:06.890766 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000506890764, "job": 14, "event": "table_file_deletion", "file_number": 4313} 2024/11/19-15:15:06.890900 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004317.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:06.890909 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000506890908, "job": 14, "event": "table_file_deletion", "file_number": 4317} 2024/11/19-15:15:06.891073 139851148035840 [compaction_job.cc:2437] [default] [JOB 15] Compacting 4@1 files to L2, score 3.25, subcompactions 1 : 1 2024/11/19-15:15:06.891082 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 12 Base level 1, inputs: [4198(31MB) 4200(31MB) 4201(31MB) 4202(31MB)] 2024/11/19-15:15:06.903368 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004334.sst): first pass time = 0.18's, 364.974'MB/sec 2024/11/19-15:15:06.903428 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004334.sst): old prealloc_size = 73819750, real_size = 65022176 2024/11/19-15:15:06.903501 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004334.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63606 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1017.06 avg-zval =1020.31 2024/11/19-15:15:06.903554 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004334.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.099 sec, Key+Value = 0.066 GB 2024/11/19-15:15:06.906632 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004335.sst): first pass time = 0.17's, 393.106'MB/sec 2024/11/19-15:15:06.906681 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004335.sst): old prealloc_size = 73819750, real_size = 65040000 2024/11/19-15:15:06.906732 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004335.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63630 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1016.95 avg-zval =1020.20 2024/11/19-15:15:06.906768 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004335.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.090 sec, Key+Value = 0.066 GB 2024/11/19-15:15:06.928244 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004333.sst): second pass time = 0.00's,150704118.881'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 933.100'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 6.026'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.295'MB/sec write SST data time = 0.02's, 4163.433'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 150704118.881'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63569 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.04 avg-zval = 1020.29 usrkeys = 63569 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.04 avg-zval = 1020.29 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0000 GB value = 0.0649 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index =3973.0625 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 0.517912 GB avg = 0.540 KB (by entry num) total key len = 0.023257 GB avg = 0.024 KB total ukey len = 0.015582 GB avg = 0.016 KB total ukey num = 0.000959424 Billion total entry num = 0.000959424 Billion write speed all = 44.366837782 MB/sec (with seq num) write speed all = 43.737583237 MB/sec (without seq num) 2024/11/19-15:15:06.932585 139851137550080 [compaction_job.cc:1948] [default] [JOB 8] Generated table #4333: 63569 keys, 64860211 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:06.932640 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000506932616, "cf_name": "default", "job": 8, "event": "table_file_creation", "file_number": 4333, "file_size": 64860211, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64858896, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1271380, "raw_average_key_size": 20, "raw_value_size": 64652067, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63569, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000506, "oldest_key_time": 0, "file_creation_time": 1732000506, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4333, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:06.940261 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004336.sst): first pass time = 0.18's, 358.040'MB/sec 2024/11/19-15:15:06.940320 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004336.sst): old prealloc_size = 73819750, real_size = 64999632 2024/11/19-15:15:06.940387 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004336.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63589 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1016.97 avg-zval =1020.23 2024/11/19-15:15:06.940441 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004336.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.099 sec, Key+Value = 0.066 GB 2024/11/19-15:15:06.952678 139851137550080 [compaction_job.cc:703] job-00008: subcompact[0], size: 0.130998 G, files: /storage/mytopling/.rocksdb [004326,004333] 2024/11/19-15:15:06.953533 139851137550080 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:15:06.953595 139851137550080 (Original Log Time 2024/11/19-15:15:06.348439) EVENT_LOG_v1 {"time_micros": 1732000506348420, "job": 8, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4182, 4183, 4185, 4186], "score": 4.00374, "input_data_size": 134100928, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:06.953601 139851137550080 (Original Log Time 2024/11/19-15:15:06.953160) [compaction_job.cc:2022] [default] [JOB 8] Compacted 4@1 files to L2 => 130997600 bytes 2024/11/19-15:15:06.953604 139851137550080 (Original Log Time 2024/11/19-15:15:06.953500) [compaction_job.cc:1224] [default] compacted to: files[4 73 66 75 0 0 0] max score 3.00 (119 files need compaction), MB/sec: 221.9 rd, 216.8 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(124.9 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128282, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:06.953611 139851137550080 (Original Log Time 2024/11/19-15:15:06.953527) EVENT_LOG_v1 {"time_micros": 1732000506953514, "cf": "default", "job": 8, "event": "compaction_finished", "compaction_time_micros": 604258, "compaction_time_cpu_micros": 536471, "output_level": 2, "num_output_files": 2, "total_output_size": 130997600, "num_input_records": 128282, "num_output_records": 128282, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 73, 66, 75, 0, 0, 0]} 2024/11/19-15:15:06.953693 139851137550080 [compaction_job.cc:2437] [default] [JOB 16] Compacting 4@1 files to L2, score 3.00, subcompactions 1 : 1 2024/11/19-15:15:06.953702 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 14 Base level 1, inputs: [4204(31MB) 4205(31MB) 4207(31MB) 4208(31MB)] 2024/11/19-15:15:06.964449 139851179493120 [dcompact_etcd.cc:1163] job-00012/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.295 sec, input = 126.976 MiB 2024/11/19-15:15:06.968543 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004335.sst): second pass time = 0.00's,193159665.672'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1051.274'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2252.379'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.354'MB/sec write SST data time = 0.02's, 3852.831'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 193159665.672'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63630 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.95 avg-zval = 1020.20 usrkeys = 63630 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.95 avg-zval = 1020.20 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7929 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0429 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 0.582620 GB avg = 0.569 KB (by entry num) total key len = 0.024530 GB avg = 0.024 KB total ukey len = 0.016345 GB avg = 0.016 KB total ukey num = 0.001023054 Billion total entry num = 0.001023054 Billion write speed all = 49.612261366 MB/sec (with seq num) write speed all = 48.943483777 MB/sec (without seq num) 2024/11/19-15:15:06.972474 139851158521600 [compaction_job.cc:1948] [default] [JOB 6] Generated table #4335: 63630 keys, 65025453 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:06.972525 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000506972500, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 4335, "file_size": 65025453, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64915504, "index_size": 108416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1272600, "raw_average_key_size": 20, "raw_value_size": 64708488, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63630, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000506, "oldest_key_time": 0, "file_creation_time": 1732000506, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4335, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:06.982189 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004331.sst): second pass time = 0.00's,166570061.538'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 376.450'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2809.855'MB/sec (index lex order gen) rebuild zvType time = 0.07's, 1.556'MB/sec write SST data time = 0.07's, 763.888'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 166570061.538'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463343 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.84 avg-zval = 117.87 usrkeys = 463343 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.84 avg-zval = 117.87 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6616 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5705 value = 0.9913 dict = 0.00 all = 1.2093 } Zip/UnZip{ index = 0.0946 value = 1.0088 dict = 0.00 all = 0.8269 } ---------------------------- total value len = 0.636755 GB avg = 0.428 KB (by entry num) total key len = 0.037503 GB avg = 0.025 KB total ukey len = 0.025612 GB avg = 0.017 KB total ukey num = 0.001486397 Billion total entry num = 0.001486397 Billion write speed all = 55.034666873 MB/sec (with seq num) write speed all = 54.064079567 MB/sec (without seq num) 2024/11/19-15:15:06.987469 139850978166528 [compaction_job.cc:1948] [default] [JOB 3] Generated table #4331: 463343 keys, 55494513 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:06.987518 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000506987493, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 4331, "file_size": 55494513, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54612384, "index_size": 876672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12973604, "raw_average_key_size": 28, "raw_value_size": 54135270, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463343, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000506, "oldest_key_time": 0, "file_creation_time": 1732000506, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4331, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:06.988168 139851158521600 [compaction_job.cc:703] job-00006: subcompact[0], size: 0.131052 G, files: /storage/mytopling/.rocksdb [004330,004335] 2024/11/19-15:15:06.993908 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004334.sst): second pass time = 0.00's,117406199.637'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 718.648'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1518.577'MB/sec (index lex order gen) rebuild zvType time = 0.07's, 0.242'MB/sec write SST data time = 0.02's, 2642.056'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 117406199.637'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63606 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.06 avg-zval = 1020.31 usrkeys = 63606 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.06 avg-zval = 1020.31 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7927 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0444 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 0.701446 GB avg = 0.453 KB (by entry num) total key len = 0.038775 GB avg = 0.025 KB total ukey len = 0.026375 GB avg = 0.017 KB total ukey num = 0.001550003 Billion total entry num = 0.001550003 Billion write speed all = 60.360970472 MB/sec (with seq num) write speed all = 59.349817228 MB/sec (without seq num) 2024/11/19-15:15:06.999941 139851127064320 [compaction_job.cc:1948] [default] [JOB 9] Generated table #4334: 63606 keys, 65007661 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:07.000008 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000506999967, "cf_name": "default", "job": 9, "event": "table_file_creation", "file_number": 4334, "file_size": 65007661, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64897744, "index_size": 108352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1272120, "raw_average_key_size": 20, "raw_value_size": 64690816, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63606, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000506, "oldest_key_time": 0, "file_creation_time": 1732000506, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4334, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:07.000208 139851158521600 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:15:07.000272 139851158521600 (Original Log Time 2024/11/19-15:15:06.389164) EVENT_LOG_v1 {"time_micros": 1732000506389143, "job": 6, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4168, 4170, 4173, 4175], "score": 4.5033, "input_data_size": 134098624, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:07.000282 139851158521600 (Original Log Time 2024/11/19-15:15:06.999809) [compaction_job.cc:2022] [default] [JOB 6] Compacted 4@1 files to L2 => 131052000 bytes 2024/11/19-15:15:07.000285 139851158521600 (Original Log Time 2024/11/19-15:15:07.000173) [compaction_job.cc:1224] [default] compacted to: files[4 69 68 75 0 0 0] max score 2.75 (117 files need compaction), MB/sec: 223.9 rd, 218.8 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128345, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:07.000291 139851158521600 (Original Log Time 2024/11/19-15:15:07.000202) EVENT_LOG_v1 {"time_micros": 1732000507000186, "cf": "default", "job": 6, "event": "compaction_finished", "compaction_time_micros": 599017, "compaction_time_cpu_micros": 536697, "output_level": 2, "num_output_files": 2, "total_output_size": 131052000, "num_input_records": 128345, "num_output_records": 128345, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 69, 68, 75, 0, 0, 0]} 2024/11/19-15:15:07.000396 139851158521600 [compaction_job.cc:2437] [default] [JOB 17] Compacting 4@1 files to L2, score 2.75, subcompactions 1 : 1 2024/11/19-15:15:07.000407 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 15 Base level 1, inputs: [4210(31MB) 4211(31MB) 4213(31MB) 4214(31MB)] 2024/11/19-15:15:07.003052 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004338.sst) /storage/mytopling/..//tmp/Topling-82038-TJM3E2 2024/11/19-15:15:07.019943 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004336.sst): second pass time = 0.00's,114254977.032'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.08's, 816.477'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1654.532'MB/sec (index lex order gen) rebuild zvType time = 0.06's, 0.276'MB/sec write SST data time = 0.02's, 2957.181'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 114254977.032'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63589 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.97 avg-zval = 1020.23 usrkeys = 63589 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.97 avg-zval = 1020.23 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7923 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0425 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 0.766114 GB avg = 0.475 KB (by entry num) total key len = 0.040047 GB avg = 0.025 KB total ukey len = 0.027139 GB avg = 0.017 KB total ukey num = 0.001613592 Billion total entry num = 0.001613592 Billion write speed all = 65.598789225 MB/sec (with seq num) write speed all = 64.548382781 MB/sec (without seq num) 2024/11/19-15:15:07.021903 139851127064320 [compaction_job.cc:703] job-00009: subcompact[0], size: 0.131145 G, files: /storage/mytopling/.rocksdb [004328,004334] 2024/11/19-15:15:07.024790 139851169007360 [compaction_job.cc:1948] [default] [JOB 5] Generated table #4336: 63589 keys, 64985069 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:07.024842 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000507024814, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 4336, "file_size": 64985069, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64875200, "index_size": 108352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1271780, "raw_average_key_size": 20, "raw_value_size": 64668317, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63589, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000506, "oldest_key_time": 0, "file_creation_time": 1732000506, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4336, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:07.025052 139851127064320 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:15:07.025121 139851127064320 (Original Log Time 2024/11/19-15:15:06.368605) EVENT_LOG_v1 {"time_micros": 1732000506368585, "job": 9, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4187, 4188, 4190, 4191], "score": 3.75396, "input_data_size": 134099328, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:07.025130 139851127064320 (Original Log Time 2024/11/19-15:15:07.024594) [compaction_job.cc:2022] [default] [JOB 9] Compacted 4@1 files to L2 => 131144922 bytes 2024/11/19-15:15:07.025135 139851127064320 (Original Log Time 2024/11/19-15:15:07.025017) [compaction_job.cc:1224] [default] compacted to: files[4 65 70 75 0 0 0] max score 2.51 (115 files need compaction), MB/sec: 205.3 rd, 200.7 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.1 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128323, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:07.025146 139851127064320 (Original Log Time 2024/11/19-15:15:07.025046) EVENT_LOG_v1 {"time_micros": 1732000507025031, "cf": "default", "job": 9, "event": "compaction_finished", "compaction_time_micros": 653320, "compaction_time_cpu_micros": 565739, "output_level": 2, "num_output_files": 2, "total_output_size": 131144922, "num_input_records": 128323, "num_output_records": 128323, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 65, 70, 75, 0, 0, 0]} 2024/11/19-15:15:07.025249 139851127064320 [compaction_job.cc:2437] [default] [JOB 18] Compacting 4@1 files to L2, score 2.51, subcompactions 1 : 1 2024/11/19-15:15:07.025262 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 16 Base level 1, inputs: [4215(31MB) 4217(31MB) 4218(31MB) 4220(31MB)] 2024/11/19-15:15:07.039677 139851169007360 [compaction_job.cc:703] job-00005: subcompact[0], size: 0.131122 G, files: /storage/mytopling/.rocksdb [004329,004336] 2024/11/19-15:15:07.040422 139851169007360 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:15:07.040477 139851169007360 (Original Log Time 2024/11/19-15:15:06.378663) EVENT_LOG_v1 {"time_micros": 1732000506378643, "job": 5, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4157, 4160, 4162, 4164], "score": 4.75308, "input_data_size": 134099648, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:07.040483 139851169007360 (Original Log Time 2024/11/19-15:15:07.040098) [compaction_job.cc:2022] [default] [JOB 5] Compacted 4@1 files to L2 => 131122074 bytes 2024/11/19-15:15:07.040486 139851169007360 (Original Log Time 2024/11/19-15:15:07.040393) [compaction_job.cc:1224] [default] compacted to: files[4 61 72 75 0 0 0] max score 2.45 (113 files need compaction), MB/sec: 202.9 rd, 198.4 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128302, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:07.040492 139851169007360 (Original Log Time 2024/11/19-15:15:07.040416) EVENT_LOG_v1 {"time_micros": 1732000507040405, "cf": "default", "job": 5, "event": "compaction_finished", "compaction_time_micros": 661028, "compaction_time_cpu_micros": 573478, "output_level": 2, "num_output_files": 2, "total_output_size": 131122074, "num_input_records": 128302, "num_output_records": 128302, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 61, 72, 75, 0, 0, 0]} 2024/11/19-15:15:07.040575 139851169007360 [compaction_job.cc:2437] [default] [JOB 19] Compacting 1@2 files to L3, score 2.45, subcompactions 1 : 1 2024/11/19-15:15:07.040583 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 17 Base level 2, inputs: [4266(3639B)] 2024/11/19-15:15:07.145656 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004338.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:15:07.177300 139851148035840 [dcompact_etcd.cc:1163] job-00015/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.286 sec, input = 126.896 MiB 2024/11/19-15:15:07.196188 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004338.sst): index pass time = 0.05's, 146.554'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462311 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:15:07.247621 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004338.sst): first pass time = 0.24's, 273.651'MB/sec 2024/11/19-15:15:07.247674 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004338.sst): old prealloc_size = 73819750, real_size = 55477616 2024/11/19-15:15:07.247906 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004338.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462311 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.76 avg-zval = 117.85 2024/11/19-15:15:07.247964 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004338.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.102 sec, Key+Value = 0.067 GB 2024/11/19-15:15:07.264844 139851137550080 [dcompact_etcd.cc:1163] job-00016/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.311 sec, input = 126.809 MiB 2024/11/19-15:15:07.289035 139851127064320 [dcompact_etcd.cc:1163] job-00018/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.263 sec, input = 126.999 MiB 2024/11/19-15:15:07.319433 139851158521600 [dcompact_etcd.cc:1163] job-00017/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.318 sec, input = 126.850 MiB 2024/11/19-15:15:07.336642 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004338.sst): second pass time = 0.00's,91798051.020'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 622.422'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3701.959'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.604'MB/sec write SST data time = 0.04's, 1248.965'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 91798051.020'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462311 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 117.85 usrkeys = 462311 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6597 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0554 GB } UnZip/Zip{ index = 10.5017 value = 0.9907 dict = 0.00 all = 1.2087 } Zip/UnZip{ index = 0.0952 value = 1.0093 dict = 0.00 all = 0.8273 } ---------------------------- total value len = 0.820092 GB avg = 0.395 KB (by entry num) total key len = 0.052992 GB avg = 0.026 KB total ukey len = 0.036385 GB avg = 0.018 KB total ukey num = 0.002075903 Billion total entry num = 0.002075903 Billion write speed all = 69.259393609 MB/sec (with seq num) write speed all = 67.941986923 MB/sec (without seq num) 2024/11/19-15:15:07.339614 139851169007360 [dcompact_etcd.cc:1163] job-00019/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.298 sec, input = 2.328 KiB 2024/11/19-15:15:07.340254 139850978166528 [compaction_job.cc:1948] [default] [JOB 3] Generated table #4338: 462311 keys, 55367281 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:07.340303 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000507340280, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 4338, "file_size": 55367281, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54481376, "index_size": 880448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12944708, "raw_average_key_size": 28, "raw_value_size": 53977254, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462311, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000507, "oldest_key_time": 0, "file_creation_time": 1732000507, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4338, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:07.353988 139850978166528 [compaction_job.cc:703] job-00003: subcompact[0], size: 0.110862 G, files: /storage/mytopling/.rocksdb [004331,004338] 2024/11/19-15:15:07.354247 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-00 2024/11/19-15:15:07.354303 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00019/att-00/compact.done) = No such file or directory: 0.006 sec on http probe failed: 1, retry 2024/11/19-15:15:07.355273 139850978166528 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:15:07.355364 139850978166528 (Original Log Time 2024/11/19-15:15:06.407959) EVENT_LOG_v1 {"time_micros": 1732000506407915, "job": 3, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4227, 4229, 4235, 4239], "score": 5.13994, "input_data_size": 133300108, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:07.355370 139850978166528 (Original Log Time 2024/11/19-15:15:07.354864) [compaction_job.cc:2022] [default] [JOB 3] Compacted 4@1 files to L2 => 110861794 bytes 2024/11/19-15:15:07.355374 139850978166528 (Original Log Time 2024/11/19-15:15:07.355243) [compaction_job.cc:1224] [default] compacted to: files[4 57 74 75 0 0 0] max score 2.53 (114 files need compaction), MB/sec: 140.9 rd, 117.2 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925654, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:07.355380 139850978166528 (Original Log Time 2024/11/19-15:15:07.355267) EVENT_LOG_v1 {"time_micros": 1732000507355255, "cf": "default", "job": 3, "event": "compaction_finished", "compaction_time_micros": 946039, "compaction_time_cpu_micros": 923667, "output_level": 2, "num_output_files": 2, "total_output_size": 110861794, "num_input_records": 925654, "num_output_records": 925654, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 57, 74, 75, 0, 0, 0]} 2024/11/19-15:15:07.355588 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003712.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:07.355603 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000507355600, "job": 3, "event": "table_file_deletion", "file_number": 3712} 2024/11/19-15:15:07.355748 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003719.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:07.355757 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000507355756, "job": 3, "event": "table_file_deletion", "file_number": 3719} 2024/11/19-15:15:07.355906 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003797.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:07.355915 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000507355914, "job": 3, "event": "table_file_deletion", "file_number": 3797} 2024/11/19-15:15:07.356088 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003960.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:07.356100 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000507356099, "job": 3, "event": "table_file_deletion", "file_number": 3960} 2024/11/19-15:15:07.356256 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004135.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:07.356266 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000507356265, "job": 3, "event": "table_file_deletion", "file_number": 4135} 2024/11/19-15:15:07.360540 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004149.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:07.360561 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000507360559, "job": 3, "event": "table_file_deletion", "file_number": 4149} 2024/11/19-15:15:07.362129 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004154.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:07.362147 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000507362145, "job": 3, "event": "table_file_deletion", "file_number": 4154} 2024/11/19-15:15:07.362512 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-00 2024/11/19-15:15:07.362563 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00019/att-00/compact.done) = No such file or directory: 0.015 sec on http probe failed: 2, retry 2024/11/19-15:15:07.365443 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004243.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:07.365462 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000507365460, "job": 3, "event": "table_file_deletion", "file_number": 4243} 2024/11/19-15:15:07.365704 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004291.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:07.365715 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000507365714, "job": 3, "event": "table_file_deletion", "file_number": 4291} 2024/11/19-15:15:07.365876 139850978166528 [compaction_job.cc:2437] [default] [JOB 20] Compacting 1@2 files to L3, score 2.53, subcompactions 1 : 1 2024/11/19-15:15:07.365885 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 18 Base level 2, inputs: [4270(32KB)] 2024/11/19-15:15:07.370281 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-00 2024/11/19-15:15:07.370328 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00019/att-00/compact.done) = No such file or directory: 0.023 sec on http probe failed: 3, retry 2024/11/19-15:15:07.377908 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-00 2024/11/19-15:15:07.377962 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00019/att-00/compact.done) = No such file or directory: 0.031 sec on http probe failed: 4, retry 2024/11/19-15:15:07.386192 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-00 2024/11/19-15:15:07.386236 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00019/att-00/compact.done) = No such file or directory: 0.038 sec on http probe failed: 5, giveup 2024/11/19-15:15:07.386250 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00019/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.345 (submit 298.496 ms wait 0.038 s), input 2.328 KiB, send shutdown 2024/11/19-15:15:07.390956 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-00 2024/11/19-15:15:07.391153 139851169007360 [dcompact_etcd.cc:1773] job-00019/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:07.690749 139850978166528 [dcompact_etcd.cc:1163] job-00020/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.324 sec, input = 24.705 KiB 2024/11/19-15:15:07.693640 139851169007360 [dcompact_etcd.cc:1163] job-00019/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.302 sec, input = 2.328 KiB 2024/11/19-15:15:07.698915 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-01 2024/11/19-15:15:07.698988 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00019/att-01/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:15:07.707008 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-01 2024/11/19-15:15:07.707057 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00019/att-01/compact.done) = No such file or directory: 0.005 sec on http probe failed: 2, retry 2024/11/19-15:15:07.712974 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-00 2024/11/19-15:15:07.713036 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00020/att-00/compact.done) = No such file or directory: 0.014 sec on http probe failed: 1, retry 2024/11/19-15:15:07.714530 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-01 2024/11/19-15:15:07.714577 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00019/att-01/compact.done) = No such file or directory: 0.014 sec on http probe failed: 3, retry 2024/11/19-15:15:07.721213 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-00 2024/11/19-15:15:07.721260 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00020/att-00/compact.done) = No such file or directory: 0.023 sec on http probe failed: 2, retry 2024/11/19-15:15:07.723104 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-01 2024/11/19-15:15:07.723150 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00019/att-01/compact.done) = No such file or directory: 0.021 sec on http probe failed: 4, retry 2024/11/19-15:15:07.726463 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-00 2024/11/19-15:15:07.726507 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00020/att-00/compact.done) = No such file or directory: 0.031 sec on http probe failed: 3, retry 2024/11/19-15:15:07.731635 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-01 2024/11/19-15:15:07.731685 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00019/att-01/compact.done) = No such file or directory: 0.030 sec on http probe failed: 5, giveup 2024/11/19-15:15:07.731699 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00019/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.340 (submit 302.419 ms wait 0.030 s), input 2.328 KiB, send shutdown 2024/11/19-15:15:07.734926 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-00 2024/11/19-15:15:07.734984 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00020/att-00/compact.done) = No such file or directory: 0.036 sec on http probe failed: 4, retry 2024/11/19-15:15:07.738224 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-01 2024/11/19-15:15:07.738417 139851169007360 [dcompact_etcd.cc:1773] job-00019/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:07.743684 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-00 2024/11/19-15:15:07.743732 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00020/att-00/compact.done) = No such file or directory: 0.045 sec on http probe failed: 5, giveup 2024/11/19-15:15:07.743746 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00020/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.377 (submit 324.422 ms wait 0.045 s), input 24.705 KiB, send shutdown 2024/11/19-15:15:07.748469 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-00 2024/11/19-15:15:07.748641 139850978166528 [dcompact_etcd.cc:1773] job-00020/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:08.017991 139851169007360 [dcompact_etcd.cc:1163] job-00019/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.279 sec, input = 2.328 KiB 2024/11/19-15:15:08.024084 139850978166528 [dcompact_etcd.cc:1163] job-00020/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.275 sec, input = 24.705 KiB 2024/11/19-15:15:08.027485 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-02 2024/11/19-15:15:08.027547 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00019/att-02/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:15:08.032788 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-02 2024/11/19-15:15:08.032833 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00019/att-02/compact.done) = No such file or directory: 0.010 sec on http probe failed: 2, retry 2024/11/19-15:15:08.040985 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-02 2024/11/19-15:15:08.041031 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00019/att-02/compact.done) = No such file or directory: 0.015 sec on http probe failed: 3, retry 2024/11/19-15:15:08.041073 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-01 2024/11/19-15:15:08.041134 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00020/att-01/compact.done) = No such file or directory: 0.011 sec on http probe failed: 1, retry 2024/11/19-15:15:08.046185 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-02 2024/11/19-15:15:08.046229 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00019/att-02/compact.done) = No such file or directory: 0.023 sec on http probe failed: 4, retry 2024/11/19-15:15:08.046469 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-01 2024/11/19-15:15:08.046513 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00020/att-01/compact.done) = No such file or directory: 0.018 sec on http probe failed: 2, retry 2024/11/19-15:15:08.050616 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-02 2024/11/19-15:15:08.050662 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00019/att-02/compact.done) = No such file or directory: 0.028 sec on http probe failed: 5, giveup 2024/11/19-15:15:08.050676 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00019/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.312 (submit 279.497 ms wait 0.028 s), input 2.328 KiB, send shutdown 2024/11/19-15:15:08.055211 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-01 2024/11/19-15:15:08.055260 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00020/att-01/compact.done) = No such file or directory: 0.023 sec on http probe failed: 3, retry 2024/11/19-15:15:08.055412 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00019/att-02 2024/11/19-15:15:08.055618 139851169007360 [dcompact_etcd.cc:1773] job-00019/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:08.057388 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00019) = success 2024/11/19-15:15:08.057399 139851169007360 [dcompact_etcd.cc:1773] job-00019/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:08.057419 139851169007360 (Original Log Time 2024/11/19-15:15:07.040595) EVENT_LOG_v1 {"time_micros": 1732000507040588, "job": 19, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4266], "score": 2.44525, "input_data_size": 3639, "oldest_snapshot_seqno": 106915316} 2024/11/19-15:15:08.057426 139851169007360 [compaction_job.cc:2437] [default] [JOB 19] Compacting 1@2 files to L3, score 2.45, subcompactions 1 : 1 2024/11/19-15:15:08.057433 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 17 Base level 2, inputs: [4266(3639B)] 2024/11/19-15:15:08.057583 139851169007360 [compaction_job.cc:703] job-00019: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:15:08.057945 139851169007360 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:15:08.058015 139851169007360 (Original Log Time 2024/11/19-15:15:08.057464) EVENT_LOG_v1 {"time_micros": 1732000508057445, "job": 19, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4266], "score": 2.44525, "input_data_size": 3639, "oldest_snapshot_seqno": 106915316} 2024/11/19-15:15:08.058021 139851169007360 (Original Log Time 2024/11/19-15:15:08.057629) [compaction_job.cc:2022] [default] [JOB 19] Compacted 1@2 files to L3 => 0 bytes 2024/11/19-15:15:08.058024 139851169007360 (Original Log Time 2024/11/19-15:15:08.057912) [compaction_job.cc:1224] [default] compacted to: files[4 57 73 75 0 0 0] max score 2.53 (113 files need compaction), MB/sec: 29.3 rd, 0.0 wr, level 3, files in(1, 0) out(0 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 18, records dropped: 18 output_compression: Snappy 2024/11/19-15:15:08.058036 139851169007360 (Original Log Time 2024/11/19-15:15:08.057940) EVENT_LOG_v1 {"time_micros": 1732000508057930, "cf": "default", "job": 19, "event": "compaction_finished", "compaction_time_micros": 124, "compaction_time_cpu_micros": 111, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 18, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 57, 73, 75, 0, 0, 0]} 2024/11/19-15:15:08.058128 139851169007360 [compaction_job.cc:2437] [default] [JOB 21] Compacting 4@2 files to L3, score 2.53, subcompactions 1 : 1 2024/11/19-15:15:08.058136 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 19 Base level 2, inputs: [4306(62MB) 4310(61MB) 4303(62MB) 4309(61MB)] 2024/11/19-15:15:08.060501 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-01 2024/11/19-15:15:08.060551 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00020/att-01/compact.done) = No such file or directory: 0.032 sec on http probe failed: 4, retry 2024/11/19-15:15:08.069663 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-01 2024/11/19-15:15:08.069711 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00020/att-01/compact.done) = No such file or directory: 0.037 sec on http probe failed: 5, giveup 2024/11/19-15:15:08.069725 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00020/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.321 (submit 275.379 ms wait 0.037 s), input 24.705 KiB, send shutdown 2024/11/19-15:15:08.075126 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-01 2024/11/19-15:15:08.075310 139850978166528 [dcompact_etcd.cc:1773] job-00020/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:08.363575 139851169007360 [dcompact_etcd.cc:1163] job-00021/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.305 sec, input = 254.477 MiB 2024/11/19-15:15:08.367425 139850978166528 [dcompact_etcd.cc:1163] job-00020/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.292 sec, input = 24.705 KiB 2024/11/19-15:15:08.373007 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-02 2024/11/19-15:15:08.373067 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00020/att-02/compact.done) = No such file or directory: 0.001 sec on http probe failed: 1, retry 2024/11/19-15:15:08.382414 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-02 2024/11/19-15:15:08.382462 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00020/att-02/compact.done) = No such file or directory: 0.006 sec on http probe failed: 2, retry 2024/11/19-15:15:08.390894 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-02 2024/11/19-15:15:08.390952 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00020/att-02/compact.done) = No such file or directory: 0.016 sec on http probe failed: 3, retry 2024/11/19-15:15:08.396467 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-02 2024/11/19-15:15:08.396513 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00020/att-02/compact.done) = No such file or directory: 0.024 sec on http probe failed: 4, retry 2024/11/19-15:15:08.401642 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-02 2024/11/19-15:15:08.401685 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00020/att-02/compact.done) = No such file or directory: 0.030 sec on http probe failed: 5, giveup 2024/11/19-15:15:08.401698 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00020/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.326 (submit 292.044 ms wait 0.030 s), input 24.705 KiB, send shutdown 2024/11/19-15:15:08.411251 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00020/att-02 2024/11/19-15:15:08.411449 139850978166528 [dcompact_etcd.cc:1773] job-00020/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:08.413160 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00020) = success 2024/11/19-15:15:08.413169 139850978166528 [dcompact_etcd.cc:1773] job-00020/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:08.413186 139850978166528 (Original Log Time 2024/11/19-15:15:07.365897) EVENT_LOG_v1 {"time_micros": 1732000507365890, "job": 20, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4270], "score": 2.52846, "input_data_size": 33540, "oldest_snapshot_seqno": 106928319} 2024/11/19-15:15:08.413193 139850978166528 [compaction_job.cc:2437] [default] [JOB 20] Compacting 1@2 files to L3, score 2.53, subcompactions 1 : 1 2024/11/19-15:15:08.413200 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 18 Base level 2, inputs: [4270(32KB)] 2024/11/19-15:15:08.413359 139850978166528 [compaction_job.cc:703] job-00020: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:15:08.413706 139850978166528 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:15:08.413763 139850978166528 (Original Log Time 2024/11/19-15:15:08.413227) EVENT_LOG_v1 {"time_micros": 1732000508413211, "job": 20, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4270], "score": 2.52846, "input_data_size": 33540, "oldest_snapshot_seqno": 106928319} 2024/11/19-15:15:08.413768 139850978166528 (Original Log Time 2024/11/19-15:15:08.413400) [compaction_job.cc:2022] [default] [JOB 20] Compacted 1@2 files to L3 => 0 bytes 2024/11/19-15:15:08.413772 139850978166528 (Original Log Time 2024/11/19-15:15:08.413682) [compaction_job.cc:1224] [default] compacted to: files[4 57 72 75 0 0 0] max score 2.36 (109 files need compaction), MB/sec: 244.8 rd, 0.0 wr, level 3, files in(1, 0) out(0 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 179, records dropped: 179 output_compression: Snappy 2024/11/19-15:15:08.413786 139850978166528 (Original Log Time 2024/11/19-15:15:08.413700) EVENT_LOG_v1 {"time_micros": 1732000508413690, "cf": "default", "job": 20, "event": "compaction_finished", "compaction_time_micros": 137, "compaction_time_cpu_micros": 124, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 179, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 57, 72, 75, 0, 0, 0]} 2024/11/19-15:15:08.413868 139850978166528 [compaction_job.cc:2437] [default] [JOB 22] Compacting 2@2 files to L3, score 2.36, subcompactions 1 : 1 2024/11/19-15:15:08.413875 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 20 Base level 2, inputs: [4300(62MB) 4301(23MB)] 2024/11/19-15:15:08.683326 139850978166528 [dcompact_etcd.cc:1163] job-00022/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.269 sec, input = 88.120 MiB 2024/11/19-15:15:10.147906 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-01 2024/11/19-15:15:10.148022 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00012/att-01/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:15:10.357517 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-00 2024/11/19-15:15:10.357618 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00015/att-00/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:15:10.443676 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-00 2024/11/19-15:15:10.443752 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00016/att-00/compact.done) = No such file or directory: 3.170 sec on http probe failed: 1, retry 2024/11/19-15:15:10.473084 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-00 2024/11/19-15:15:10.473152 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00018/att-00/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:15:10.499108 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-00 2024/11/19-15:15:10.499177 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00017/att-00/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:15:10.892571 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-00 2024/11/19-15:15:10.892655 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00022/att-00/compact.done) = No such file or directory: 2.203 sec on http probe failed: 1, retry 2024/11/19-15:15:12.156839 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-01 2024/11/19-15:15:12.156970 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00012/att-01/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:15:12.367119 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-00 2024/11/19-15:15:12.367198 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00015/att-00/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:15:12.452611 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-00 2024/11/19-15:15:12.452713 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00016/att-00/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:15:12.481591 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-00 2024/11/19-15:15:12.481667 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00018/att-00/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:15:12.508771 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-00 2024/11/19-15:15:12.508912 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00017/att-00/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:15:12.664135 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-00 2024/11/19-15:15:12.664211 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00022/att-00/compact.done) = No such file or directory: 3.972 sec on http probe failed: 2, retry 2024/11/19-15:15:13.373144 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-00 2024/11/19-15:15:13.373267 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00021/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:15:14.162724 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-01 2024/11/19-15:15:14.162855 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00012/att-01/compact.done) = No such file or directory: 7.193 sec on http probe failed: 3, retry 2024/11/19-15:15:14.372042 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-00 2024/11/19-15:15:14.372161 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00015/att-00/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:15:14.435639 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-00 2024/11/19-15:15:14.435730 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00022/att-00/compact.done) = No such file or directory: 5.743 sec on http probe failed: 3, retry 2024/11/19-15:15:14.458196 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-00 2024/11/19-15:15:14.458287 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00016/att-00/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:15:14.491582 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-00 2024/11/19-15:15:14.491670 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00018/att-00/compact.done) = No such file or directory: 7.193 sec on http probe failed: 3, retry 2024/11/19-15:15:14.516557 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-00 2024/11/19-15:15:14.516713 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00017/att-00/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:15:15.382656 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-00 2024/11/19-15:15:15.382757 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00021/att-00/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:15:16.171752 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-01 2024/11/19-15:15:16.171885 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00012/att-01/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:15:16.203807 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-00 2024/11/19-15:15:16.203881 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00022/att-00/compact.done) = No such file or directory: 7.515 sec on http probe failed: 4, retry 2024/11/19-15:15:16.376954 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-00 2024/11/19-15:15:16.377029 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00015/att-00/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:15:16.466168 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-00 2024/11/19-15:15:16.466259 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00016/att-00/compact.done) = No such file or directory: 9.194 sec on http probe failed: 4, retry 2024/11/19-15:15:16.499864 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-00 2024/11/19-15:15:16.499974 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00018/att-00/compact.done) = No such file or directory: 9.203 sec on http probe failed: 4, retry 2024/11/19-15:15:16.525051 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-00 2024/11/19-15:15:16.525169 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00017/att-00/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:15:16.635664 139852349699840 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:15:17.388266 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-00 2024/11/19-15:15:17.388353 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00021/att-00/compact.done) = No such file or directory: 9.019 sec on http probe failed: 3, retry 2024/11/19-15:15:17.974029 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-00 2024/11/19-15:15:17.974129 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00022/att-00/compact.done) = No such file or directory: 9.283 sec on http probe failed: 5, giveup 2024/11/19-15:15:17.974147 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00022/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 88.120 sec, real 9.560 (submit 269.052 ms wait 9.283 s), input 88.120 MiB, send shutdown 2024/11/19-15:15:17.981646 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-00 2024/11/19-15:15:17.981877 139850978166528 [dcompact_etcd.cc:1773] job-00022/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:18.176866 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-01 2024/11/19-15:15:18.176974 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00012/att-01/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:15:18.177001 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00012/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.976 sec, real 11.508 (submit 295.103 ms wait 11.208 s), input 126.976 MiB, send shutdown 2024/11/19-15:15:18.185708 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-01 2024/11/19-15:15:18.186006 139851179493120 [dcompact_etcd.cc:1773] job-00012/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:18.309450 139850978166528 [dcompact_etcd.cc:1163] job-00022/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.327 sec, input = 88.120 MiB 2024/11/19-15:15:18.381815 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-00 2024/11/19-15:15:18.381878 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00015/att-00/compact.done) = No such file or directory: 11.200 sec on http probe failed: 5, giveup 2024/11/19-15:15:18.381895 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00015/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.896 sec, real 11.490 (submit 285.781 ms wait 11.200 s), input 126.896 MiB, send shutdown 2024/11/19-15:15:18.390623 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-00 2024/11/19-15:15:18.390824 139851148035840 [dcompact_etcd.cc:1773] job-00015/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:18.451322 139851179493120 [dcompact_etcd.cc:1163] job-00012/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.265 sec, input = 126.976 MiB 2024/11/19-15:15:18.474583 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-00 2024/11/19-15:15:18.474636 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00016/att-00/compact.done) = No such file or directory: 11.201 sec on http probe failed: 5, giveup 2024/11/19-15:15:18.474651 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00016/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.809 sec, real 11.520 (submit 310.607 ms wait 11.201 s), input 126.809 MiB, send shutdown 2024/11/19-15:15:18.483258 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-00 2024/11/19-15:15:18.483457 139851137550080 [dcompact_etcd.cc:1773] job-00016/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:18.505035 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-00 2024/11/19-15:15:18.505099 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00018/att-00/compact.done) = No such file or directory: 11.211 sec on http probe failed: 5, giveup 2024/11/19-15:15:18.505116 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00018/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.999 sec, real 11.479 (submit 263.181 ms wait 11.211 s), input 126.999 MiB, send shutdown 2024/11/19-15:15:18.510574 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-00 2024/11/19-15:15:18.510803 139851127064320 [dcompact_etcd.cc:1773] job-00018/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:18.529970 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-00 2024/11/19-15:15:18.530049 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00017/att-00/compact.done) = No such file or directory: 11.206 sec on http probe failed: 5, giveup 2024/11/19-15:15:18.530074 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00017/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.850 sec, real 11.529 (submit 318.354 ms wait 11.206 s), input 126.850 MiB, send shutdown 2024/11/19-15:15:18.534503 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-00 2024/11/19-15:15:18.534738 139851158521600 [dcompact_etcd.cc:1773] job-00017/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:18.680006 139851148035840 [dcompact_etcd.cc:1163] job-00015/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 126.896 MiB 2024/11/19-15:15:18.746488 139851137550080 [dcompact_etcd.cc:1163] job-00016/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.263 sec, input = 126.809 MiB 2024/11/19-15:15:18.811130 139851127064320 [dcompact_etcd.cc:1163] job-00018/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.300 sec, input = 126.999 MiB 2024/11/19-15:15:18.821177 139851158521600 [dcompact_etcd.cc:1163] job-00017/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.286 sec, input = 126.850 MiB 2024/11/19-15:15:19.398230 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-00 2024/11/19-15:15:19.398339 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00021/att-00/compact.done) = No such file or directory: 11.025 sec on http probe failed: 4, retry 2024/11/19-15:15:20.517367 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-01 2024/11/19-15:15:20.517466 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00022/att-01/compact.done) = No such file or directory: 2.203 sec on http probe failed: 1, retry 2024/11/19-15:15:21.404206 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-00 2024/11/19-15:15:21.404298 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00021/att-00/compact.done) = No such file or directory: 13.035 sec on http probe failed: 5, giveup 2024/11/19-15:15:21.404315 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00021/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 254.477 sec, real 13.346 (submit 304.959 ms wait 13.035 s), input 254.477 MiB, send shutdown 2024/11/19-15:15:21.412568 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-00 2024/11/19-15:15:21.412790 139851169007360 [dcompact_etcd.cc:1773] job-00021/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:21.635471 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-02 2024/11/19-15:15:21.635611 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00012/att-02/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:15:21.730989 139851169007360 [dcompact_etcd.cc:1163] job-00021/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.318 sec, input = 254.477 MiB 2024/11/19-15:15:21.860515 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-01 2024/11/19-15:15:21.860631 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00015/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:15:21.922036 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-01 2024/11/19-15:15:21.922129 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00016/att-01/compact.done) = No such file or directory: 3.170 sec on http probe failed: 1, retry 2024/11/19-15:15:21.994555 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-01 2024/11/19-15:15:21.994635 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00018/att-01/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:15:22.000948 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-01 2024/11/19-15:15:22.001014 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00017/att-01/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:15:22.287336 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-01 2024/11/19-15:15:22.287409 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00022/att-01/compact.done) = No such file or directory: 3.970 sec on http probe failed: 2, retry 2024/11/19-15:15:23.468549 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4339. Immutable memtables: 0. 2024/11/19-15:15:23.468768 139852362282752 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:15:23.469047 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 23] Syncing log #4324 2024/11/19-15:15:23.469147 139850957195008 (Original Log Time 2024/11/19-15:15:23.468813) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:15:23.469157 139850957195008 [flush_job.cc:879] [default] [JOB 23] Flushing memtable with next log file: 4339 2024/11/19-15:15:23.469177 139850957195008 EVENT_LOG_v1 {"time_micros": 1732000523469171, "job": 23, "event": "flush_started", "num_memtables": 1, "num_entries": 1393203, "num_deletes": 0, "total_data_size": 473489077, "memory_usage": 538181000, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:15:23.469181 139850957195008 [flush_job.cc:910] [default] [JOB 23] Level-0 flush table #4340: started 2024/11/19-15:15:23.469576 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000001.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004340.sst in ConvertToSST, mem_size = 513.249 M, time = 386.676 us 2024/11/19-15:15:23.645361 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-02 2024/11/19-15:15:23.645502 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00012/att-02/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:15:23.869978 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-01 2024/11/19-15:15:23.870136 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00015/att-01/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:15:23.930402 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-01 2024/11/19-15:15:23.930529 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00016/att-01/compact.done) = No such file or directory: 5.176 sec on http probe failed: 2, retry 2024/11/19-15:15:24.003760 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-01 2024/11/19-15:15:24.003818 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00018/att-01/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:15:24.006305 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004340.sst): fsize = 513.251 M, time(ms): open: 0.037, seek: 0.024, finish: 0.048, meta: 0.012, Flush: 0.010, sync: 536.572, close: 0.010, all: 536.713 2024/11/19-15:15:24.007453 139850957195008 [flush_job.cc:1083] [default] [JOB 23] Flush lasted 538306 microseconds, and 399537 cpu microseconds. 2024/11/19-15:15:24.009469 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-01 2024/11/19-15:15:24.009522 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00017/att-01/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:15:24.028014 139850957195008 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:15:24.028091 139850957195008 (Original Log Time 2024/11/19-15:15:24.006343) [flush_job.cc:1029] [default] [JOB 23] Level-0 flush table #4340: 538182426 bytes OK (needs compaction) 2024/11/19-15:15:24.028096 139850957195008 (Original Log Time 2024/11/19-15:15:24.007468) [memtable_list.cc:560] [default] Level-0 commit table #4340 started 2024/11/19-15:15:24.028099 139850957195008 (Original Log Time 2024/11/19-15:15:24.027977) [memtable_list.cc:759] [default] Level-0 commit table #4340: memtable #1 done 2024/11/19-15:15:24.028102 139850957195008 (Original Log Time 2024/11/19-15:15:24.028005) EVENT_LOG_v1 {"time_micros": 1732000524027997, "job": 23, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [5, 57, 72, 75, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:15:24.028105 139850957195008 (Original Log Time 2024/11/19-15:15:24.028066) [db_impl_compaction_flush.cc:310] [default] Level summary: files[5 57 72 75 0 0 0] max score 2.31 (108 files need compaction) 2024/11/19-15:15:24.054550 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-01 2024/11/19-15:15:24.054651 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00022/att-01/compact.done) = No such file or directory: 5.740 sec on http probe failed: 3, retry 2024/11/19-15:15:25.652187 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-02 2024/11/19-15:15:25.652276 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00012/att-02/compact.done) = No such file or directory: 7.194 sec on http probe failed: 3, retry 2024/11/19-15:15:25.825561 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-01 2024/11/19-15:15:25.825659 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00022/att-01/compact.done) = No such file or directory: 7.508 sec on http probe failed: 4, retry 2024/11/19-15:15:25.878734 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-01 2024/11/19-15:15:25.878830 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00015/att-01/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:15:25.935941 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-01 2024/11/19-15:15:25.936037 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00016/att-01/compact.done) = No such file or directory: 7.184 sec on http probe failed: 3, retry 2024/11/19-15:15:26.011390 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-01 2024/11/19-15:15:26.011475 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00018/att-01/compact.done) = No such file or directory: 7.193 sec on http probe failed: 3, retry 2024/11/19-15:15:26.018856 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-01 2024/11/19-15:15:26.018947 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00017/att-01/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:15:26.741012 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-01 2024/11/19-15:15:26.741118 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00021/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:15:27.593719 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-01 2024/11/19-15:15:27.593816 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00022/att-01/compact.done) = No such file or directory: 9.279 sec on http probe failed: 5, giveup 2024/11/19-15:15:27.593834 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00022/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 88.120 sec, real 9.612 (submit 327.483 ms wait 9.279 s), input 88.120 MiB, send shutdown 2024/11/19-15:15:27.602846 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-01 2024/11/19-15:15:27.603073 139850978166528 [dcompact_etcd.cc:1773] job-00022/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:27.661537 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-02 2024/11/19-15:15:27.661640 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00012/att-02/compact.done) = No such file or directory: 9.201 sec on http probe failed: 4, retry 2024/11/19-15:15:27.887880 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-01 2024/11/19-15:15:27.888016 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00015/att-01/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:15:27.908431 139850978166528 [dcompact_etcd.cc:1163] job-00022/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.305 sec, input = 88.120 MiB 2024/11/19-15:15:27.944863 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-01 2024/11/19-15:15:27.944941 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00016/att-01/compact.done) = No such file or directory: 9.190 sec on http probe failed: 4, retry 2024/11/19-15:15:28.018956 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-01 2024/11/19-15:15:28.019013 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00018/att-01/compact.done) = No such file or directory: 9.200 sec on http probe failed: 4, retry 2024/11/19-15:15:28.023447 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-01 2024/11/19-15:15:28.023502 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00017/att-01/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:15:28.750647 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-01 2024/11/19-15:15:28.750723 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00021/att-01/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:15:29.671724 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-02 2024/11/19-15:15:29.671803 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00012/att-02/compact.done) = No such file or directory: 11.210 sec on http probe failed: 5, giveup 2024/11/19-15:15:29.671822 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00012/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.976 sec, real 11.486 (submit 265.204 ms wait 11.210 s), input 126.976 MiB, send shutdown 2024/11/19-15:15:29.679927 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00012/att-02 2024/11/19-15:15:29.680155 139851179493120 [dcompact_etcd.cc:1773] job-00012/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:29.682085 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00012) = success 2024/11/19-15:15:29.682092 139851179493120 [dcompact_etcd.cc:1773] job-00012/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:29.682112 139851179493120 (Original Log Time 2024/11/19-15:14:55.124306) EVENT_LOG_v1 {"time_micros": 1732000495124295, "job": 12, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4193, 4194, 4196, 4197], "score": 3.50418, "input_data_size": 134099904, "oldest_snapshot_seqno": 106066230} 2024/11/19-15:15:29.682120 139851179493120 [compaction_job.cc:2437] [default] [JOB 12] Compacting 4@1 files to L2, score 3.50, subcompactions 1 : 1 2024/11/19-15:15:29.682128 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 10 Base level 1, inputs: [4193(31MB) 4194(31MB) 4196(31MB) 4197(31MB)] 2024/11/19-15:15:29.682355 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004341.sst) /storage/mytopling/..//tmp/Topling-82038-VgN9kR 2024/11/19-15:15:29.736359 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004341.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:15:29.736460 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004341.sst): index pass time = 0.00's,7322.584'MB/sec index type = UintIndex_AllOne usrkeys = 64717 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:15:29.802448 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004341.sst): first pass time = 0.12's, 558.876'MB/sec 2024/11/19-15:15:29.802497 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004341.sst): old prealloc_size = 73819750, real_size = 66041880 2024/11/19-15:15:29.802520 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004341.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64717 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.96 avg-zval =1020.22 2024/11/19-15:15:29.855663 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004341.sst): second pass time = 0.00's,151997281.755'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1242.300'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.925'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.419'MB/sec write SST data time = 0.01's, 4554.401'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 151997281.755'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64717 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.96 avg-zval = 1020.22 usrkeys = 64717 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.96 avg-zval = 1020.22 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.8125 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 0.885907 GB avg = 0.414 KB (by entry num) total key len = 0.054286 GB avg = 0.025 KB total ukey len = 0.037161 GB avg = 0.017 KB total ukey num = 0.002140620 Billion total entry num = 0.002140620 Billion write speed all = 26.767042825 MB/sec (with seq num) write speed all = 26.279499715 MB/sec (without seq num) 2024/11/19-15:15:29.859290 139851179493120 [compaction_job.cc:1948] [default] [JOB 12] Generated table #4341: 64717 keys, 66026675 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:29.859342 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000529859319, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 4341, "file_size": 66026675, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025376, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294340, "raw_average_key_size": 20, "raw_value_size": 65814823, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64717, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000529, "oldest_key_time": 0, "file_creation_time": 1732000529, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4341, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:29.873584 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004342.sst) /storage/mytopling/..//tmp/Topling-82038-2BDYYF 2024/11/19-15:15:29.893821 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-01 2024/11/19-15:15:29.893915 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00015/att-01/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:15:29.893952 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00015/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.896 sec, real 11.503 (submit 289.101 ms wait 11.208 s), input 126.896 MiB, send shutdown 2024/11/19-15:15:29.901496 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-01 2024/11/19-15:15:29.901764 139851148035840 [dcompact_etcd.cc:1773] job-00015/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:29.927005 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004342.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:15:29.930265 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004342.sst): index pass time = 0.00's, 157.380'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63679 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:15:29.954907 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-01 2024/11/19-15:15:29.955028 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00016/att-01/compact.done) = No such file or directory: 11.199 sec on http probe failed: 5, giveup 2024/11/19-15:15:29.955053 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00016/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.809 sec, real 11.472 (submit 262.951 ms wait 11.199 s), input 126.809 MiB, send shutdown 2024/11/19-15:15:29.963221 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-01 2024/11/19-15:15:29.963468 139851137550080 [dcompact_etcd.cc:1773] job-00016/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:29.992723 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004342.sst): first pass time = 0.12's, 554.348'MB/sec 2024/11/19-15:15:29.992777 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004342.sst): old prealloc_size = 73819750, real_size = 65093424 2024/11/19-15:15:29.992836 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004342.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63679 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0648 GB zip-val = 0.0650 GB avg-val =1017.00 avg-zval =1020.26 2024/11/19-15:15:29.992875 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004342.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.066 GB 2024/11/19-15:15:30.025168 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-01 2024/11/19-15:15:30.025260 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00018/att-01/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:15:30.025279 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00018/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.999 sec, real 11.514 (submit 300.244 ms wait 11.208 s), input 126.999 MiB, send shutdown 2024/11/19-15:15:30.032254 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-01 2024/11/19-15:15:30.032313 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00017/att-01/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:15:30.032328 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00017/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.850 sec, real 11.497 (submit 286.323 ms wait 11.202 s), input 126.850 MiB, send shutdown 2024/11/19-15:15:30.033577 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-01 2024/11/19-15:15:30.033797 139851127064320 [dcompact_etcd.cc:1773] job-00018/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:30.036687 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-01 2024/11/19-15:15:30.036876 139851158521600 [dcompact_etcd.cc:1773] job-00017/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:30.051273 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004342.sst): second pass time = 0.00's,90829851.332'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1112.632'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2055.636'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.377'MB/sec write SST data time = 0.02's, 4026.100'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 90829851.332'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63679 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.26 usrkeys = 63679 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7935 UnZipSize{ index = 0.0008 GB value = 0.0648 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0650 GB dict = 0.00 MB all = 0.0651 GB } UnZip/Zip{ index = 7.0441 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 0.950668 GB avg = 0.431 KB (by entry num) total key len = 0.055560 GB avg = 0.025 KB total ukey len = 0.037926 GB avg = 0.017 KB total ukey num = 0.002204299 Billion total entry num = 0.002204299 Billion write speed all = 28.488394241 MB/sec (with seq num) write speed all = 27.989128206 MB/sec (without seq num) 2024/11/19-15:15:30.054848 139851179493120 [compaction_job.cc:1948] [default] [JOB 12] Generated table #4342: 63679 keys, 65078893 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:30.054895 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000530054872, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 4342, "file_size": 65078893, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64968864, "index_size": 108480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1273580, "raw_average_key_size": 20, "raw_value_size": 64761684, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63679, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000529, "oldest_key_time": 0, "file_creation_time": 1732000529, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4342, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:30.068960 139851179493120 [compaction_job.cc:703] job-00012: subcompact[0], size: 0.131106 G, files: /storage/mytopling/.rocksdb [004341,004342] 2024/11/19-15:15:30.069740 139851179493120 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:15:30.069866 139851179493120 (Original Log Time 2024/11/19-15:15:29.682159) EVENT_LOG_v1 {"time_micros": 1732000529682140, "job": 12, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4193, 4194, 4196, 4197], "score": 3.50418, "input_data_size": 134099904, "oldest_snapshot_seqno": 106066230} 2024/11/19-15:15:30.069872 139851179493120 (Original Log Time 2024/11/19-15:15:30.069374) [compaction_job.cc:2022] [default] [JOB 12] Compacted 4@1 files to L2 => 131105568 bytes 2024/11/19-15:15:30.069876 139851179493120 (Original Log Time 2024/11/19-15:15:30.069704) [compaction_job.cc:1224] [default] compacted to: files[5 53 74 75 0 0 0] max score 2.39 (110 files need compaction), MB/sec: 346.7 rd, 338.9 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128396, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:30.069892 139851179493120 (Original Log Time 2024/11/19-15:15:30.069731) EVENT_LOG_v1 {"time_micros": 1732000530069717, "cf": "default", "job": 12, "event": "compaction_finished", "compaction_time_micros": 386812, "compaction_time_cpu_micros": 384983, "output_level": 2, "num_output_files": 2, "total_output_size": 131105568, "num_input_records": 128396, "num_output_records": 128396, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 53, 74, 75, 0, 0, 0]} 2024/11/19-15:15:30.073835 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004176.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:30.073861 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000530073858, "job": 12, "event": "table_file_deletion", "file_number": 4176} 2024/11/19-15:15:30.077951 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004178.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:30.077974 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000530077971, "job": 12, "event": "table_file_deletion", "file_number": 4178} 2024/11/19-15:15:30.082136 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004179.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:30.082158 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000530082156, "job": 12, "event": "table_file_deletion", "file_number": 4179} 2024/11/19-15:15:30.086250 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004181.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:30.086270 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000530086268, "job": 12, "event": "table_file_deletion", "file_number": 4181} 2024/11/19-15:15:30.086438 139851179493120 [compaction_job.cc:2437] [default] [JOB 24] Compacting 2@2 files to L3, score 2.39, subcompactions 1 : 1 2024/11/19-15:15:30.086447 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 22 Base level 2, inputs: [4302(13MB) 4290(16MB)] 2024/11/19-15:15:30.119855 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-02 2024/11/19-15:15:30.119940 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00022/att-02/compact.done) = No such file or directory: 2.203 sec on http probe failed: 1, retry 2024/11/19-15:15:30.191683 139851148035840 [dcompact_etcd.cc:1163] job-00015/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.290 sec, input = 126.896 MiB 2024/11/19-15:15:30.274250 139851137550080 [dcompact_etcd.cc:1163] job-00016/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.311 sec, input = 126.809 MiB 2024/11/19-15:15:30.359646 139851158521600 [dcompact_etcd.cc:1163] job-00017/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.323 sec, input = 126.850 MiB 2024/11/19-15:15:30.361400 139851127064320 [dcompact_etcd.cc:1163] job-00018/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.328 sec, input = 126.999 MiB 2024/11/19-15:15:30.399698 139851179493120 [dcompact_etcd.cc:1163] job-00024/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.313 sec, input = 37.892 MiB 2024/11/19-15:15:30.756304 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-01 2024/11/19-15:15:30.756380 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00021/att-01/compact.done) = No such file or directory: 9.020 sec on http probe failed: 3, retry 2024/11/19-15:15:31.355453 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-00 2024/11/19-15:15:31.355551 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00024/att-00/compact.done) = No such file or directory: 0.947 sec on http probe failed: 1, retry 2024/11/19-15:15:31.889788 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-02 2024/11/19-15:15:31.889861 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00022/att-02/compact.done) = No such file or directory: 3.974 sec on http probe failed: 2, retry 2024/11/19-15:15:32.119383 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-00 2024/11/19-15:15:32.119462 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00024/att-00/compact.done) = No such file or directory: 1.714 sec on http probe failed: 2, retry 2024/11/19-15:15:32.766134 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-01 2024/11/19-15:15:32.766228 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00021/att-01/compact.done) = No such file or directory: 11.025 sec on http probe failed: 4, retry 2024/11/19-15:15:32.885488 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-00 2024/11/19-15:15:32.885583 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00024/att-00/compact.done) = No such file or directory: 2.478 sec on http probe failed: 3, retry 2024/11/19-15:15:33.372548 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-02 2024/11/19-15:15:33.372644 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00015/att-02/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:15:33.452820 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-02 2024/11/19-15:15:33.452888 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00016/att-02/compact.done) = No such file or directory: 3.170 sec on http probe failed: 1, retry 2024/11/19-15:15:33.536097 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-02 2024/11/19-15:15:33.536157 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00017/att-02/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:15:33.541794 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-02 2024/11/19-15:15:33.541855 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00018/att-02/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:15:33.651119 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-00 2024/11/19-15:15:33.651201 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00024/att-00/compact.done) = No such file or directory: 3.244 sec on http probe failed: 4, retry 2024/11/19-15:15:33.660116 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-02 2024/11/19-15:15:33.660176 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00022/att-02/compact.done) = No such file or directory: 5.744 sec on http probe failed: 3, retry 2024/11/19-15:15:34.414724 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-00 2024/11/19-15:15:34.414816 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00024/att-00/compact.done) = No such file or directory: 4.009 sec on http probe failed: 5, giveup 2024/11/19-15:15:34.414834 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00024/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 37.892 sec, real 4.328 (submit 312.776 ms wait 4.009 s), input 37.892 MiB, send shutdown 2024/11/19-15:15:34.422104 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-00 2024/11/19-15:15:34.422348 139851179493120 [dcompact_etcd.cc:1773] job-00024/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:34.716750 139851179493120 [dcompact_etcd.cc:1163] job-00024/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.294 sec, input = 37.892 MiB 2024/11/19-15:15:34.775447 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-01 2024/11/19-15:15:34.775511 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00021/att-01/compact.done) = No such file or directory: 13.035 sec on http probe failed: 5, giveup 2024/11/19-15:15:34.775528 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00021/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 254.477 sec, real 13.363 (submit 318.107 ms wait 13.035 s), input 254.477 MiB, send shutdown 2024/11/19-15:15:34.784758 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-01 2024/11/19-15:15:34.784982 139851169007360 [dcompact_etcd.cc:1773] job-00021/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:35.059424 139851169007360 [dcompact_etcd.cc:1163] job-00021/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.274 sec, input = 254.477 MiB 2024/11/19-15:15:35.380728 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-02 2024/11/19-15:15:35.380803 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00015/att-02/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:15:35.431291 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-02 2024/11/19-15:15:35.431349 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00022/att-02/compact.done) = No such file or directory: 7.514 sec on http probe failed: 4, retry 2024/11/19-15:15:35.461862 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-02 2024/11/19-15:15:35.461917 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00016/att-02/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:15:35.544103 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-02 2024/11/19-15:15:35.544172 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00017/att-02/compact.done) = No such file or directory: 5.177 sec on http probe failed: 2, retry 2024/11/19-15:15:35.550884 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-02 2024/11/19-15:15:35.550955 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00018/att-02/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:15:35.669428 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-01 2024/11/19-15:15:35.669504 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00024/att-01/compact.done) = No such file or directory: 0.947 sec on http probe failed: 1, retry 2024/11/19-15:15:36.433863 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-01 2024/11/19-15:15:36.433969 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00024/att-01/compact.done) = No such file or directory: 1.711 sec on http probe failed: 2, retry 2024/11/19-15:15:37.199057 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-02 2024/11/19-15:15:37.199132 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00022/att-02/compact.done) = No such file or directory: 9.285 sec on http probe failed: 5, giveup 2024/11/19-15:15:37.199150 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00022/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 88.120 sec, real 9.596 (submit 305.278 ms wait 9.285 s), input 88.120 MiB, send shutdown 2024/11/19-15:15:37.199809 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-01 2024/11/19-15:15:37.199864 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00024/att-01/compact.done) = No such file or directory: 2.475 sec on http probe failed: 3, retry 2024/11/19-15:15:37.207031 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00022/att-02 2024/11/19-15:15:37.207246 139850978166528 [dcompact_etcd.cc:1773] job-00022/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:37.209101 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00022) = success 2024/11/19-15:15:37.209108 139850978166528 [dcompact_etcd.cc:1773] job-00022/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:37.209126 139850978166528 (Original Log Time 2024/11/19-15:15:08.413885) EVENT_LOG_v1 {"time_micros": 1732000508413880, "job": 22, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4300, 4301], "score": 2.36277, "input_data_size": 90139443, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:37.209133 139850978166528 [compaction_job.cc:2437] [default] [JOB 22] Compacting 2@2 files to L3, score 2.36, subcompactions 1 : 1 2024/11/19-15:15:37.209140 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 20 Base level 2, inputs: [4300(62MB) 4301(23MB)] 2024/11/19-15:15:37.224530 139850978166528 [compaction_job.cc:703] job-00022: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:15:37.224930 139850978166528 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:15:37.225042 139850978166528 (Original Log Time 2024/11/19-15:15:37.209170) EVENT_LOG_v1 {"time_micros": 1732000537209153, "job": 22, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4300, 4301], "score": 2.36277, "input_data_size": 90139443, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:37.225049 139850978166528 (Original Log Time 2024/11/19-15:15:37.224593) [compaction_job.cc:2022] [default] [JOB 22] Compacted 2@2 files to L3 => 0 bytes 2024/11/19-15:15:37.225052 139850978166528 (Original Log Time 2024/11/19-15:15:37.224893) [compaction_job.cc:1224] [default] compacted to: files[5 53 72 75 0 0 0] max score 2.36 (108 files need compaction), MB/sec: 5863.9 rd, 0.0 wr, level 3, files in(2, 0) out(0 +0 blob) MB in(86.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 138124, records dropped: 138124 output_compression: Snappy 2024/11/19-15:15:37.225069 139850978166528 (Original Log Time 2024/11/19-15:15:37.224914) EVENT_LOG_v1 {"time_micros": 1732000537224902, "cf": "default", "job": 22, "event": "compaction_finished", "compaction_time_micros": 15372, "compaction_time_cpu_micros": 15288, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 138124, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 53, 72, 75, 0, 0, 0]} 2024/11/19-15:15:37.225169 139850978166528 [compaction_job.cc:2437] [default] [JOB 25] Compacting 1@2 files to L3, score 2.36, subcompactions 1 : 1 2024/11/19-15:15:37.225177 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 23 Base level 2, inputs: [4292(13MB)] 2024/11/19-15:15:37.389671 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-02 2024/11/19-15:15:37.389741 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00015/att-02/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:15:37.469531 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-02 2024/11/19-15:15:37.469587 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00016/att-02/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:15:37.505807 139850978166528 [dcompact_etcd.cc:1163] job-00025/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 22.431 MiB 2024/11/19-15:15:37.550521 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-02 2024/11/19-15:15:37.550582 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00017/att-02/compact.done) = No such file or directory: 7.185 sec on http probe failed: 3, retry 2024/11/19-15:15:37.557804 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-02 2024/11/19-15:15:37.557872 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00018/att-02/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:15:37.966327 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-01 2024/11/19-15:15:37.966408 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00024/att-01/compact.done) = No such file or directory: 3.241 sec on http probe failed: 4, retry 2024/11/19-15:15:37.967302 139852349699840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16273899 2024/11/19-15:15:38.075254 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-00 2024/11/19-15:15:38.075338 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00025/att-00/compact.done) = No such file or directory: 0.561 sec on http probe failed: 1, retry 2024/11/19-15:15:38.533040 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-00 2024/11/19-15:15:38.533205 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00025/att-00/compact.done) = No such file or directory: 1.018 sec on http probe failed: 2, retry 2024/11/19-15:15:38.729554 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-01 2024/11/19-15:15:38.729642 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00024/att-01/compact.done) = No such file or directory: 4.008 sec on http probe failed: 5, giveup 2024/11/19-15:15:38.729660 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00024/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 37.892 sec, real 4.307 (submit 294.311 ms wait 4.008 s), input 37.892 MiB, send shutdown 2024/11/19-15:15:38.737153 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-01 2024/11/19-15:15:38.746282 139851179493120 [dcompact_etcd.cc:1773] job-00024/att-01: CleanFiles 0.009 sec : 0.000 2024/11/19-15:15:38.990543 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-00 2024/11/19-15:15:38.990645 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00025/att-00/compact.done) = No such file or directory: 1.476 sec on http probe failed: 3, retry 2024/11/19-15:15:39.003214 139851179493120 [dcompact_etcd.cc:1163] job-00024/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.257 sec, input = 37.892 MiB 2024/11/19-15:15:39.398200 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-02 2024/11/19-15:15:39.398276 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00015/att-02/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:15:39.448396 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-00 2024/11/19-15:15:39.448454 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00025/att-00/compact.done) = No such file or directory: 1.934 sec on http probe failed: 4, retry 2024/11/19-15:15:39.477653 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-02 2024/11/19-15:15:39.477703 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00016/att-02/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:15:39.555281 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-02 2024/11/19-15:15:39.555334 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00017/att-02/compact.done) = No such file or directory: 9.191 sec on http probe failed: 4, retry 2024/11/19-15:15:39.565356 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-02 2024/11/19-15:15:39.565415 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00018/att-02/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:15:39.902468 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-00 2024/11/19-15:15:39.902544 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00025/att-00/compact.done) = No such file or directory: 2.391 sec on http probe failed: 5, giveup 2024/11/19-15:15:39.902561 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00025/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 22.431 sec, real 2.677 (submit 280.213 ms wait 2.391 s), input 22.431 MiB, send shutdown 2024/11/19-15:15:39.910981 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-00 2024/11/19-15:15:39.911210 139850978166528 [dcompact_etcd.cc:1773] job-00025/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:39.960935 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-02 2024/11/19-15:15:39.961029 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00024/att-02/compact.done) = No such file or directory: 0.947 sec on http probe failed: 1, retry 2024/11/19-15:15:40.064320 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-02 2024/11/19-15:15:40.064400 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00021/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:15:40.178657 139850978166528 [dcompact_etcd.cc:1163] job-00025/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.267 sec, input = 22.431 MiB 2024/11/19-15:15:40.724827 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-02 2024/11/19-15:15:40.724906 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00024/att-02/compact.done) = No such file or directory: 1.716 sec on http probe failed: 2, retry 2024/11/19-15:15:40.745828 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-01 2024/11/19-15:15:40.745894 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00025/att-01/compact.done) = No such file or directory: 0.561 sec on http probe failed: 1, retry 2024/11/19-15:15:41.199986 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-01 2024/11/19-15:15:41.200060 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00025/att-01/compact.done) = No such file or directory: 1.016 sec on http probe failed: 2, retry 2024/11/19-15:15:41.403205 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-02 2024/11/19-15:15:41.403274 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00015/att-02/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:15:41.403292 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00015/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.896 sec, real 11.501 (submit 289.809 ms wait 11.207 s), input 126.896 MiB, send shutdown 2024/11/19-15:15:41.411382 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00015/att-02 2024/11/19-15:15:41.411589 139851148035840 [dcompact_etcd.cc:1773] job-00015/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:41.413447 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00015) = success 2024/11/19-15:15:41.413457 139851148035840 [dcompact_etcd.cc:1773] job-00015/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:41.413477 139851148035840 (Original Log Time 2024/11/19-15:15:06.891094) EVENT_LOG_v1 {"time_micros": 1732000506891088, "job": 15, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4198, 4200, 4201, 4202], "score": 3.2544, "input_data_size": 134098688, "oldest_snapshot_seqno": 106912315} 2024/11/19-15:15:41.413484 139851148035840 [compaction_job.cc:2437] [default] [JOB 15] Compacting 4@1 files to L2, score 3.25, subcompactions 1 : 1 2024/11/19-15:15:41.413491 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 12 Base level 1, inputs: [4198(31MB) 4200(31MB) 4201(31MB) 4202(31MB)] 2024/11/19-15:15:41.413709 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004343.sst) /storage/mytopling/..//tmp/Topling-82038-mg4jGk 2024/11/19-15:15:41.467492 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004343.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:15:41.470840 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004343.sst): index pass time = 0.00's, 156.330'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64714 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:15:41.482991 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-02 2024/11/19-15:15:41.483114 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00016/att-02/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:15:41.483140 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00016/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.809 sec, real 11.520 (submit 310.690 ms wait 11.204 s), input 126.809 MiB, send shutdown 2024/11/19-15:15:41.491064 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-02 2024/11/19-15:15:41.491080 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00016/att-02 2024/11/19-15:15:41.491143 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00024/att-02/compact.done) = No such file or directory: 2.480 sec on http probe failed: 3, retry 2024/11/19-15:15:41.491357 139851137550080 [dcompact_etcd.cc:1773] job-00016/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:41.493253 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00016) = success 2024/11/19-15:15:41.493267 139851137550080 [dcompact_etcd.cc:1773] job-00016/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:41.493298 139851137550080 (Original Log Time 2024/11/19-15:15:06.953713) EVENT_LOG_v1 {"time_micros": 1732000506953707, "job": 16, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4204, 4205, 4207, 4208], "score": 3.00462, "input_data_size": 134100416, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:41.493328 139851137550080 [compaction_job.cc:2437] [default] [JOB 16] Compacting 4@1 files to L2, score 3.00, subcompactions 1 : 1 2024/11/19-15:15:41.493341 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 14 Base level 1, inputs: [4204(31MB) 4205(31MB) 4207(31MB) 4208(31MB)] 2024/11/19-15:15:41.493676 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004344.sst) /storage/mytopling/..//tmp/Topling-82038-0VMCwG 2024/11/19-15:15:41.546334 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004343.sst): first pass time = 0.13's, 506.058'MB/sec 2024/11/19-15:15:41.546393 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004343.sst): old prealloc_size = 73819750, real_size = 66152608 2024/11/19-15:15:41.546452 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004343.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64714 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.02 avg-zval =1020.27 2024/11/19-15:15:41.546502 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004343.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.079 sec, Key+Value = 0.067 GB 2024/11/19-15:15:41.563295 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004344.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:15:41.563928 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-02 2024/11/19-15:15:41.564056 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00017/att-02/compact.done) = No such file or directory: 11.196 sec on http probe failed: 5, giveup 2024/11/19-15:15:41.564083 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00017/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.850 sec, real 11.527 (submit 322.686 ms wait 11.196 s), input 126.850 MiB, send shutdown 2024/11/19-15:15:41.568641 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004344.sst): index pass time = 0.01's, 97.801'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64714 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:15:41.572643 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00017/att-02 2024/11/19-15:15:41.572931 139851158521600 [dcompact_etcd.cc:1773] job-00017/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:41.574186 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-02 2024/11/19-15:15:41.574258 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00018/att-02/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:15:41.574281 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00018/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.999 sec, real 11.540 (submit 327.523 ms wait 11.204 s), input 126.999 MiB, send shutdown 2024/11/19-15:15:41.574832 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00017) = success 2024/11/19-15:15:41.574848 139851158521600 [dcompact_etcd.cc:1773] job-00017/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:41.574876 139851158521600 (Original Log Time 2024/11/19-15:15:07.000429) EVENT_LOG_v1 {"time_micros": 1732000507000415, "job": 17, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4210, 4211, 4213, 4214], "score": 2.75484, "input_data_size": 134099648, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:41.574887 139851158521600 [compaction_job.cc:2437] [default] [JOB 17] Compacting 4@1 files to L2, score 2.75, subcompactions 1 : 1 2024/11/19-15:15:41.574899 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 15 Base level 1, inputs: [4210(31MB) 4211(31MB) 4213(31MB) 4214(31MB)] 2024/11/19-15:15:41.575204 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004345.sst) /storage/mytopling/..//tmp/Topling-82038-KOEwEj 2024/11/19-15:15:41.582375 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00018/att-02 2024/11/19-15:15:41.582594 139851127064320 [dcompact_etcd.cc:1773] job-00018/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:41.584436 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00018) = success 2024/11/19-15:15:41.584454 139851127064320 [dcompact_etcd.cc:1773] job-00018/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:41.584483 139851127064320 (Original Log Time 2024/11/19-15:15:07.025277) EVENT_LOG_v1 {"time_micros": 1732000507025270, "job": 18, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4215, 4217, 4218, 4220], "score": 2.50506, "input_data_size": 134098752, "oldest_snapshot_seqno": 106915316} 2024/11/19-15:15:41.584495 139851127064320 [compaction_job.cc:2437] [default] [JOB 18] Compacting 4@1 files to L2, score 2.51, subcompactions 1 : 1 2024/11/19-15:15:41.584509 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 16 Base level 1, inputs: [4215(31MB) 4217(31MB) 4218(31MB) 4220(31MB)] 2024/11/19-15:15:41.584809 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004346.sst) /storage/mytopling/..//tmp/Topling-82038-MVh9Pv 2024/11/19-15:15:41.621682 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004343.sst): second pass time = 0.00's,209603573.248'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.08's, 878.796'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2042.213'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.305'MB/sec write SST data time = 0.02's, 2984.943'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 209603573.248'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64714 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.02 avg-zval = 1020.27 usrkeys = 64714 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.02 avg-zval = 1020.27 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0464 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 1.016484 GB avg = 0.448 KB (by entry num) total key len = 0.056854 GB avg = 0.025 KB total ukey len = 0.038702 GB avg = 0.017 KB total ukey num = 0.002269013 Billion total entry num = 0.002269013 Billion write speed all = 22.890053938 MB/sec (with seq num) write speed all = 22.502941324 MB/sec (without seq num) 2024/11/19-15:15:41.627009 139851148035840 [compaction_job.cc:1948] [default] [JOB 15] Generated table #4343: 64714 keys, 66137837 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:41.627081 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000541627047, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 4343, "file_size": 66137837, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66026064, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294280, "raw_average_key_size": 20, "raw_value_size": 65815522, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64714, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000541, "oldest_key_time": 0, "file_creation_time": 1732000541, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4343, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:41.649831 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004347.sst) /storage/mytopling/..//tmp/Topling-82038-qGZYgv 2024/11/19-15:15:41.650881 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004344.sst): first pass time = 0.16's, 426.936'MB/sec 2024/11/19-15:15:41.650955 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004344.sst): old prealloc_size = 73819750, real_size = 66151728 2024/11/19-15:15:41.651025 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004344.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64714 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:15:41.651087 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004344.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.088 sec, Key+Value = 0.067 GB 2024/11/19-15:15:41.653943 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-01 2024/11/19-15:15:41.654049 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00025/att-01/compact.done) = No such file or directory: 1.470 sec on http probe failed: 3, retry 2024/11/19-15:15:41.656082 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004345.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:15:41.656182 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004345.sst): index pass time = 0.00's,6910.518'MB/sec index type = UintIndex_AllOne usrkeys = 64717 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:15:41.672488 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004346.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:15:41.672598 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004346.sst): index pass time = 0.00's,6415.724'MB/sec index type = UintIndex_AllOne usrkeys = 64713 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:15:41.737210 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004347.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:15:41.737327 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004347.sst): index pass time = 0.00's,6231.356'MB/sec index type = UintIndex_AllOne usrkeys = 63598 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:15:41.741064 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004344.sst): second pass time = 0.00's,213683935.065'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 734.053'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1662.488'MB/sec (index lex order gen) rebuild zvType time = 0.07's, 0.247'MB/sec write SST data time = 0.02's, 2703.324'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 213683935.065'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64714 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 64714 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0464 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 1.082298 GB avg = 0.464 KB (by entry num) total key len = 0.058148 GB avg = 0.025 KB total ukey len = 0.039479 GB avg = 0.017 KB total ukey num = 0.002333727 Billion total entry num = 0.002333727 Billion write speed all = 24.259457365 MB/sec (with seq num) write speed all = 23.862315128 MB/sec (without seq num) 2024/11/19-15:15:41.746741 139851137550080 [compaction_job.cc:1948] [default] [JOB 16] Generated table #4344: 64714 keys, 66136941 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:41.746814 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000541746779, "cf_name": "default", "job": 16, "event": "table_file_creation", "file_number": 4344, "file_size": 66136941, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025184, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294280, "raw_average_key_size": 20, "raw_value_size": 65814652, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64714, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000541, "oldest_key_time": 0, "file_creation_time": 1732000541, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4344, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:41.751252 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004345.sst): first pass time = 0.18's, 381.247'MB/sec 2024/11/19-15:15:41.751318 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004345.sst): old prealloc_size = 73819750, real_size = 66042440 2024/11/19-15:15:41.751349 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004345.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64717 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.97 avg-zval =1020.23 2024/11/19-15:15:41.764650 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004346.sst): first pass time = 0.18's, 373.197'MB/sec 2024/11/19-15:15:41.764703 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004346.sst): old prealloc_size = 73819750, real_size = 66042520 2024/11/19-15:15:41.764730 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004346.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64713 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.04 avg-zval =1020.29 2024/11/19-15:15:41.770463 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004348.sst) /storage/mytopling/..//tmp/Topling-82038-Ng27y2 2024/11/19-15:15:41.827113 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004345.sst): second pass time = 0.00's,152703897.912'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.08's, 871.205'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 7.007'MB/sec (index lex order gen) rebuild zvType time = 0.06's, 0.278'MB/sec write SST data time = 0.02's, 3749.981'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 152703897.912'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64717 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.97 avg-zval = 1020.23 usrkeys = 64717 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.97 avg-zval = 1020.23 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.8125 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 1.148114 GB avg = 0.479 KB (by entry num) total key len = 0.059443 GB avg = 0.025 KB total ukey len = 0.040255 GB avg = 0.017 KB total ukey num = 0.002398444 Billion total entry num = 0.002398444 Billion write speed all = 25.640068569 MB/sec (with seq num) write speed all = 25.232658954 MB/sec (without seq num) 2024/11/19-15:15:41.828375 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004347.sst): first pass time = 0.18's, 369.431'MB/sec 2024/11/19-15:15:41.832063 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004347.sst): old prealloc_size = 73819750, real_size = 64901240 2024/11/19-15:15:41.832100 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004347.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63598 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1016.98 avg-zval =1020.24 2024/11/19-15:15:41.832300 139851158521600 [compaction_job.cc:1948] [default] [JOB 17] Generated table #4345: 64717 keys, 66027251 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:41.832363 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000541832339, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 4345, "file_size": 66027251, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025936, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294340, "raw_average_key_size": 20, "raw_value_size": 65815380, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64717, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000541, "oldest_key_time": 0, "file_creation_time": 1732000541, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4345, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:41.840536 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004346.sst): second pass time = 0.00's,134317297.959'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.08's, 870.978'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 7.688'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.296'MB/sec write SST data time = 0.02's, 3132.280'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 134317297.959'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64713 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.04 avg-zval = 1020.29 usrkeys = 64713 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.04 avg-zval = 1020.29 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.5625 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 1.213929 GB avg = 0.493 KB (by entry num) total key len = 0.060737 GB avg = 0.025 KB total ukey len = 0.041032 GB avg = 0.017 KB total ukey num = 0.002463157 Billion total entry num = 0.002463157 Billion write speed all = 27.057304112 MB/sec (with seq num) write speed all = 26.639021217 MB/sec (without seq num) 2024/11/19-15:15:41.846064 139851127064320 [compaction_job.cc:1948] [default] [JOB 18] Generated table #4346: 64713 keys, 66027315 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:41.846129 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000541846096, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 4346, "file_size": 66027315, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66026016, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294260, "raw_average_key_size": 20, "raw_value_size": 65815476, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64713, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000541, "oldest_key_time": 0, "file_creation_time": 1732000541, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4346, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:41.849750 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004349.sst) /storage/mytopling/..//tmp/Topling-82038-SSXWKs 2024/11/19-15:15:41.852589 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004348.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:15:41.855875 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004348.sst): index pass time = 0.00's, 155.706'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63511 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:15:41.870041 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004350.sst) /storage/mytopling/..//tmp/Topling-82038-sK9Z0g 2024/11/19-15:15:41.886222 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004347.sst): second pass time = 0.00's,175279390.244'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1122.851'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 6.716'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.403'MB/sec write SST data time = 0.02's, 3542.058'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 175279390.244'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63598 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.98 avg-zval = 1020.24 usrkeys = 63598 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.98 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0000 GB value = 0.0649 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index =3974.8750 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 1.278607 GB avg = 0.506 KB (by entry num) total key len = 0.062009 GB avg = 0.025 KB total ukey len = 0.041795 GB avg = 0.017 KB total ukey num = 0.002526755 Billion total entry num = 0.002526755 Billion write speed all = 28.429645650 MB/sec (with seq num) write speed all = 28.000978643 MB/sec (without seq num) 2024/11/19-15:15:41.890128 139851148035840 [compaction_job.cc:1948] [default] [JOB 15] Generated table #4347: 63598 keys, 64886259 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:41.890180 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000541890156, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 4347, "file_size": 64886259, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64884992, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1271960, "raw_average_key_size": 20, "raw_value_size": 64678095, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63598, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000541, "oldest_key_time": 0, "file_creation_time": 1732000541, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4347, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:41.904167 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004349.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:15:41.904721 139851148035840 [compaction_job.cc:703] job-00015: subcompact[0], size: 0.131024 G, files: /storage/mytopling/.rocksdb [004343,004347] 2024/11/19-15:15:41.905507 139851148035840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777215 2024/11/19-15:15:41.905645 139851148035840 (Original Log Time 2024/11/19-15:15:41.413520) EVENT_LOG_v1 {"time_micros": 1732000541413504, "job": 15, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4198, 4200, 4201, 4202], "score": 3.2544, "input_data_size": 134098688, "oldest_snapshot_seqno": 106912315} 2024/11/19-15:15:41.905652 139851148035840 (Original Log Time 2024/11/19-15:15:41.905147) [compaction_job.cc:2022] [default] [JOB 15] Compacted 4@1 files to L2 => 131024096 bytes 2024/11/19-15:15:41.905655 139851148035840 (Original Log Time 2024/11/19-15:15:41.905474) [compaction_job.cc:1224] [default] compacted to: files[5 49 74 75 0 0 0] max score 2.43 (109 files need compaction), MB/sec: 273.0 rd, 266.7 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128312, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:41.905671 139851148035840 (Original Log Time 2024/11/19-15:15:41.905500) EVENT_LOG_v1 {"time_micros": 1732000541905488, "cf": "default", "job": 15, "event": "compaction_finished", "compaction_time_micros": 491213, "compaction_time_cpu_micros": 485103, "output_level": 2, "num_output_files": 2, "total_output_size": 131024096, "num_input_records": 128312, "num_output_records": 128312, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 49, 74, 75, 0, 0, 0]} 2024/11/19-15:15:41.909526 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004182.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:41.909555 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000541909552, "job": 15, "event": "table_file_deletion", "file_number": 4182} 2024/11/19-15:15:41.910781 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004349.sst): index pass time = 0.01's, 77.226'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63550 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:15:41.913597 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004183.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:41.913622 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000541913619, "job": 15, "event": "table_file_deletion", "file_number": 4183} 2024/11/19-15:15:41.917433 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004185.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:41.917454 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000541917452, "job": 15, "event": "table_file_deletion", "file_number": 4185} 2024/11/19-15:15:41.921365 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004186.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:41.921385 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000541921382, "job": 15, "event": "table_file_deletion", "file_number": 4186} 2024/11/19-15:15:41.921581 139851148035840 [compaction_job.cc:2437] [default] [JOB 26] Compacting 4@2 files to L3, score 2.43, subcompactions 1 : 1 2024/11/19-15:15:41.921590 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 24 Base level 2, inputs: [4293(26MB) 4304(52MB) 4312(52MB) 4305(52MB)] 2024/11/19-15:15:41.943725 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004348.sst): first pass time = 0.17's, 380.165'MB/sec 2024/11/19-15:15:41.943786 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004348.sst): old prealloc_size = 73819750, real_size = 64920240 2024/11/19-15:15:41.943856 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004348.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63511 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0646 GB zip-val = 0.0648 GB avg-val =1016.98 avg-zval =1020.23 2024/11/19-15:15:41.943930 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004348.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.091 sec, Key+Value = 0.066 GB 2024/11/19-15:15:41.956137 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004350.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:15:41.961061 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004350.sst): index pass time = 0.00's, 104.175'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63699 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:15:41.972511 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004349.sst): first pass time = 0.12's, 536.901'MB/sec 2024/11/19-15:15:41.972557 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004349.sst): old prealloc_size = 73819750, real_size = 64962176 2024/11/19-15:15:41.972612 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004349.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63550 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0646 GB zip-val = 0.0648 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:15:41.972657 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004349.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.068 sec, Key+Value = 0.066 GB 2024/11/19-15:15:42.035420 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004349.sst): second pass time = 0.00's,185189240.688'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1033.467'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2152.501'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.371'MB/sec write SST data time = 0.02's, 3244.305'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 185189240.688'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63550 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 63550 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7919 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0648 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index = 7.0423 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 1.343238 GB avg = 0.519 KB (by entry num) total key len = 0.063280 GB avg = 0.024 KB total ukey len = 0.042558 GB avg = 0.016 KB total ukey num = 0.002590305 Billion total entry num = 0.002590305 Billion write speed all = 29.733119198 MB/sec (with seq num) write speed all = 29.295056827 MB/sec (without seq num) 2024/11/19-15:15:42.035436 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004348.sst): second pass time = 0.00's,155636862.651'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 708.368'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1690.921'MB/sec (index lex order gen) rebuild zvType time = 0.06's, 0.247'MB/sec write SST data time = 0.03's, 2385.943'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 155636862.651'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63511 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 1020.23 usrkeys = 63511 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 1020.23 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7914 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0648 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index = 7.0422 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 1.407828 GB avg = 0.530 KB (by entry num) total key len = 0.064550 GB avg = 0.024 KB total ukey len = 0.043320 GB avg = 0.016 KB total ukey num = 0.002653816 Billion total entry num = 0.002653816 Billion write speed all = 31.125355111 MB/sec (with seq num) write speed all = 30.676552041 MB/sec (without seq num) 2024/11/19-15:15:42.039436 139851158521600 [compaction_job.cc:1948] [default] [JOB 17] Generated table #4349: 63550 keys, 64947629 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:42.039485 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000542039462, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 4349, "file_size": 64947629, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64837808, "index_size": 108288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1271000, "raw_average_key_size": 20, "raw_value_size": 64631045, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63550, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000541, "oldest_key_time": 0, "file_creation_time": 1732000541, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4349, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:42.044673 139851137550080 [compaction_job.cc:1948] [default] [JOB 16] Generated table #4348: 63511 keys, 64905709 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:42.044733 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000542044702, "cf_name": "default", "job": 16, "event": "table_file_creation", "file_number": 4348, "file_size": 64905709, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64795936, "index_size": 108224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1270220, "raw_average_key_size": 20, "raw_value_size": 64589298, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63511, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000541, "oldest_key_time": 0, "file_creation_time": 1732000541, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4348, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:42.048751 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004350.sst): first pass time = 0.18's, 369.692'MB/sec 2024/11/19-15:15:42.048795 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004350.sst): old prealloc_size = 73819750, real_size = 65115856 2024/11/19-15:15:42.048849 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004350.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63699 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0648 GB zip-val = 0.0650 GB avg-val =1017.03 avg-zval =1020.29 2024/11/19-15:15:42.048897 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004350.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.093 sec, Key+Value = 0.066 GB 2024/11/19-15:15:42.054753 139851158521600 [compaction_job.cc:703] job-00017: subcompact[0], size: 0.130975 G, files: /storage/mytopling/.rocksdb [004345,004349] 2024/11/19-15:15:42.055590 139851158521600 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:15:42.055707 139851158521600 (Original Log Time 2024/11/19-15:15:41.574953) EVENT_LOG_v1 {"time_micros": 1732000541574915, "job": 17, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4210, 4211, 4213, 4214], "score": 2.75484, "input_data_size": 134099648, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:42.055714 139851158521600 (Original Log Time 2024/11/19-15:15:42.055230) [compaction_job.cc:2022] [default] [JOB 17] Compacted 4@1 files to L2 => 130974880 bytes 2024/11/19-15:15:42.055717 139851158521600 (Original Log Time 2024/11/19-15:15:42.055558) [compaction_job.cc:1224] [default] compacted to: files[5 45 76 75 0 0 0] max score 2.37 (107 files need compaction), MB/sec: 279.5 rd, 273.0 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(124.9 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128267, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:42.055733 139851158521600 (Original Log Time 2024/11/19-15:15:42.055583) EVENT_LOG_v1 {"time_micros": 1732000542055571, "cf": "default", "job": 17, "event": "compaction_finished", "compaction_time_micros": 479813, "compaction_time_cpu_micros": 472436, "output_level": 2, "num_output_files": 2, "total_output_size": 130974880, "num_input_records": 128267, "num_output_records": 128267, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 45, 76, 75, 0, 0, 0]} 2024/11/19-15:15:42.055830 139851158521600 [compaction_job.cc:2437] [default] [JOB 27] Compacting 4@2 files to L3, score 2.37, subcompactions 1 : 1 2024/11/19-15:15:42.055840 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 25 Base level 2, inputs: [4311(52MB) 4308(52MB) 4314(52MB) 4307(52MB)] 2024/11/19-15:15:42.059494 139851137550080 [compaction_job.cc:703] job-00016: subcompact[0], size: 0.131043 G, files: /storage/mytopling/.rocksdb [004344,004348] 2024/11/19-15:15:42.060199 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:15:42.060292 139851137550080 (Original Log Time 2024/11/19-15:15:41.493392) EVENT_LOG_v1 {"time_micros": 1732000541493361, "job": 16, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4204, 4205, 4207, 4208], "score": 3.00462, "input_data_size": 134100416, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:42.060313 139851137550080 (Original Log Time 2024/11/19-15:15:42.059881) [compaction_job.cc:2022] [default] [JOB 16] Compacted 4@1 files to L2 => 131042650 bytes 2024/11/19-15:15:42.060317 139851137550080 (Original Log Time 2024/11/19-15:15:42.060171) [compaction_job.cc:1224] [default] compacted to: files[5 41 78 75 0 0 0] max score 2.28 (105 files need compaction), MB/sec: 236.9 rd, 231.5 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128225, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:42.060336 139851137550080 (Original Log Time 2024/11/19-15:15:42.060192) EVENT_LOG_v1 {"time_micros": 1732000542060181, "cf": "default", "job": 16, "event": "compaction_finished", "compaction_time_micros": 566114, "compaction_time_cpu_micros": 556601, "output_level": 2, "num_output_files": 2, "total_output_size": 131042650, "num_input_records": 128225, "num_output_records": 128225, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 41, 78, 75, 0, 0, 0]} 2024/11/19-15:15:42.064195 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004168.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.064218 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000542064215, "job": 16, "event": "table_file_deletion", "file_number": 4168} 2024/11/19-15:15:42.068529 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004170.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.068552 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000542068549, "job": 16, "event": "table_file_deletion", "file_number": 4170} 2024/11/19-15:15:42.070425 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-02 2024/11/19-15:15:42.070502 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00021/att-02/compact.done) = No such file or directory: 7.005 sec on http probe failed: 2, retry 2024/11/19-15:15:42.072967 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004173.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.072989 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000542072987, "job": 16, "event": "table_file_deletion", "file_number": 4173} 2024/11/19-15:15:42.077077 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004175.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.077097 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000542077095, "job": 16, "event": "table_file_deletion", "file_number": 4175} 2024/11/19-15:15:42.081093 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004187.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.081112 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000542081110, "job": 16, "event": "table_file_deletion", "file_number": 4187} 2024/11/19-15:15:42.085229 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004188.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.085249 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000542085247, "job": 16, "event": "table_file_deletion", "file_number": 4188} 2024/11/19-15:15:42.089382 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004190.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.089403 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000542089401, "job": 16, "event": "table_file_deletion", "file_number": 4190} 2024/11/19-15:15:42.093415 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004191.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.093434 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000542093432, "job": 16, "event": "table_file_deletion", "file_number": 4191} 2024/11/19-15:15:42.093610 139851137550080 [compaction_job.cc:2437] [default] [JOB 28] Compacting 4@2 files to L3, score 2.28, subcompactions 1 : 1 2024/11/19-15:15:42.093619 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 26 Base level 2, inputs: [4315(52MB) 4331(52MB) 4338(52MB) 4322(52MB)] 2024/11/19-15:15:42.107505 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004350.sst): second pass time = 0.00's,206318675.159'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1109.027'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2133.627'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.371'MB/sec write SST data time = 0.02's, 4137.347'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 206318675.159'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63699 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 1020.29 usrkeys = 63699 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 1020.29 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7937 UnZipSize{ index = 0.0008 GB value = 0.0648 GB dict = 0.00 MB all = 0.0661 GB } __ZipSize{ index = 0.0001 GB value = 0.0650 GB dict = 0.00 MB all = 0.0651 GB } UnZip/Zip{ index = 7.0463 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 1.472612 GB avg = 0.542 KB (by entry num) total key len = 0.065824 GB avg = 0.024 KB total ukey len = 0.044084 GB avg = 0.016 KB total ukey num = 0.002717515 Billion total entry num = 0.002717515 Billion write speed all = 32.472304186 MB/sec (with seq num) write speed all = 32.013427890 MB/sec (without seq num) 2024/11/19-15:15:42.108228 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-01 2024/11/19-15:15:42.108286 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00025/att-01/compact.done) = No such file or directory: 1.924 sec on http probe failed: 4, retry 2024/11/19-15:15:42.111106 139851127064320 [compaction_job.cc:1948] [default] [JOB 18] Generated table #4350: 63699 keys, 65101293 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:42.111147 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000542111125, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 4350, "file_size": 65101293, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64991296, "index_size": 108480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1273980, "raw_average_key_size": 20, "raw_value_size": 64784064, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63699, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000541, "oldest_key_time": 0, "file_creation_time": 1732000541, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4350, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:42.125108 139851127064320 [compaction_job.cc:703] job-00018: subcompact[0], size: 0.131129 G, files: /storage/mytopling/.rocksdb [004346,004350] 2024/11/19-15:15:42.125775 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:15:42.125880 139851127064320 (Original Log Time 2024/11/19-15:15:41.584553) EVENT_LOG_v1 {"time_micros": 1732000541584528, "job": 18, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4215, 4217, 4218, 4220], "score": 2.50506, "input_data_size": 134098752, "oldest_snapshot_seqno": 106915316} 2024/11/19-15:15:42.125887 139851127064320 (Original Log Time 2024/11/19-15:15:42.125473) [compaction_job.cc:2022] [default] [JOB 18] Compacted 4@1 files to L2 => 131128608 bytes 2024/11/19-15:15:42.125890 139851127064320 (Original Log Time 2024/11/19-15:15:42.125749) [compaction_job.cc:1224] [default] compacted to: files[5 37 80 75 0 0 0] max score 2.26 (103 files need compaction), MB/sec: 248.1 rd, 242.6 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.1 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128412, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:42.125904 139851127064320 (Original Log Time 2024/11/19-15:15:42.125769) EVENT_LOG_v1 {"time_micros": 1732000542125759, "cf": "default", "job": 18, "event": "compaction_finished", "compaction_time_micros": 540565, "compaction_time_cpu_micros": 534041, "output_level": 2, "num_output_files": 2, "total_output_size": 131128608, "num_input_records": 128412, "num_output_records": 128412, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 37, 80, 75, 0, 0, 0]} 2024/11/19-15:15:42.129699 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004157.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.129720 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000542129717, "job": 18, "event": "table_file_deletion", "file_number": 4157} 2024/11/19-15:15:42.133514 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004160.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.133534 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000542133532, "job": 18, "event": "table_file_deletion", "file_number": 4160} 2024/11/19-15:15:42.137369 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004162.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.137388 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000542137386, "job": 18, "event": "table_file_deletion", "file_number": 4162} 2024/11/19-15:15:42.141356 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004164.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.141374 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000542141372, "job": 18, "event": "table_file_deletion", "file_number": 4164} 2024/11/19-15:15:42.145377 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004227.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.145396 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000542145394, "job": 18, "event": "table_file_deletion", "file_number": 4227} 2024/11/19-15:15:42.150201 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004229.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.150219 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000542150217, "job": 18, "event": "table_file_deletion", "file_number": 4229} 2024/11/19-15:15:42.154195 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004235.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.154214 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000542154212, "job": 18, "event": "table_file_deletion", "file_number": 4235} 2024/11/19-15:15:42.158063 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004239.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.158081 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000542158079, "job": 18, "event": "table_file_deletion", "file_number": 4239} 2024/11/19-15:15:42.158232 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004266.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:42.158241 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000542158240, "job": 18, "event": "table_file_deletion", "file_number": 4266} 2024/11/19-15:15:42.158443 139851127064320 [compaction_job.cc:2437] [default] [JOB 29] Compacting 1@2 files to L3, score 2.20, subcompactions 1 : 1 2024/11/19-15:15:42.158451 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 27 Base level 2, inputs: [4323(5283KB)] 2024/11/19-15:15:42.232018 139851148035840 [dcompact_etcd.cc:1163] job-00026/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.310 sec, input = 223.779 MiB 2024/11/19-15:15:42.256140 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-02 2024/11/19-15:15:42.256201 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00024/att-02/compact.done) = No such file or directory: 3.246 sec on http probe failed: 4, retry 2024/11/19-15:15:42.375034 139851158521600 [dcompact_etcd.cc:1163] job-00027/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.319 sec, input = 255.637 MiB 2024/11/19-15:15:42.411851 139851137550080 [dcompact_etcd.cc:1163] job-00028/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.318 sec, input = 255.641 MiB 2024/11/19-15:15:42.445751 139851127064320 [dcompact_etcd.cc:1163] job-00029/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.287 sec, input = 6.230 MiB 2024/11/19-15:15:42.565450 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-01 2024/11/19-15:15:42.565561 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00025/att-01/compact.done) = No such file or directory: 2.378 sec on http probe failed: 5, giveup 2024/11/19-15:15:42.565587 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00025/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 22.431 sec, real 2.654 (submit 267.368 ms wait 2.378 s), input 22.431 MiB, send shutdown 2024/11/19-15:15:42.574088 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-01 2024/11/19-15:15:42.574341 139850978166528 [dcompact_etcd.cc:1773] job-00025/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:42.610750 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-00 2024/11/19-15:15:42.610817 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00029/att-00/compact.done) = No such file or directory: 0.156 sec on http probe failed: 1, retry 2024/11/19-15:15:42.740398 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-00 2024/11/19-15:15:42.740463 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00029/att-00/compact.done) = No such file or directory: 0.290 sec on http probe failed: 2, retry 2024/11/19-15:15:42.862291 139850978166528 [dcompact_etcd.cc:1163] job-00025/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.288 sec, input = 22.431 MiB 2024/11/19-15:15:42.871521 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-00 2024/11/19-15:15:42.871581 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00029/att-00/compact.done) = No such file or directory: 0.419 sec on http probe failed: 3, retry 2024/11/19-15:15:43.005735 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-00 2024/11/19-15:15:43.005816 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00029/att-00/compact.done) = No such file or directory: 0.550 sec on http probe failed: 4, retry 2024/11/19-15:15:43.022219 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-02 2024/11/19-15:15:43.022270 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00024/att-02/compact.done) = No such file or directory: 4.011 sec on http probe failed: 5, giveup 2024/11/19-15:15:43.022289 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00024/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 37.892 sec, real 4.276 (submit 256.831 ms wait 4.011 s), input 37.892 MiB, send shutdown 2024/11/19-15:15:43.031082 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00024/att-02 2024/11/19-15:15:43.031300 139851179493120 [dcompact_etcd.cc:1773] job-00024/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:43.033121 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00024) = success 2024/11/19-15:15:43.033129 139851179493120 [dcompact_etcd.cc:1773] job-00024/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:43.033148 139851179493120 (Original Log Time 2024/11/19-15:15:30.086460) EVENT_LOG_v1 {"time_micros": 1732000530086453, "job": 24, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4302, 4290], "score": 2.38806, "input_data_size": 32120834, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:43.033155 139851179493120 [compaction_job.cc:2437] [default] [JOB 24] Compacting 2@2 files to L3, score 2.39, subcompactions 1 : 1 2024/11/19-15:15:43.033162 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 22 Base level 2, inputs: [4302(13MB) 4290(16MB)] 2024/11/19-15:15:43.076407 139851179493120 [compaction_job.cc:703] job-00024: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:15:43.076792 139851179493120 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:15:43.076895 139851179493120 (Original Log Time 2024/11/19-15:15:43.033191) EVENT_LOG_v1 {"time_micros": 1732000543033175, "job": 24, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4302, 4290], "score": 2.38806, "input_data_size": 32120834, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:43.076905 139851179493120 (Original Log Time 2024/11/19-15:15:43.076468) [compaction_job.cc:2022] [default] [JOB 24] Compacted 2@2 files to L3 => 0 bytes 2024/11/19-15:15:43.076910 139851179493120 (Original Log Time 2024/11/19-15:15:43.076763) [compaction_job.cc:1224] [default] compacted to: files[5 37 78 75 0 0 0] max score 2.20 (62 files need compaction), MB/sec: 743.0 rd, 0.0 wr, level 3, files in(2, 0) out(0 +0 blob) MB in(30.6, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 537000, records dropped: 537000 output_compression: Snappy 2024/11/19-15:15:43.076946 139851179493120 (Original Log Time 2024/11/19-15:15:43.076786) EVENT_LOG_v1 {"time_micros": 1732000543076775, "cf": "default", "job": 24, "event": "compaction_finished", "compaction_time_micros": 43229, "compaction_time_cpu_micros": 43073, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 537000, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 37, 78, 75, 0, 0, 0]} 2024/11/19-15:15:43.089965 139851179493120 (Original Log Time 2024/11/19-15:15:43.076974) [db_impl_compaction_flush.cc:3374] [default] Manual compaction from level-0 to level-1 from '00000131' seq:72057594037927935, type:22 .. '00000132' seq:0, type:0; will stop at (end) 2024/11/19-15:15:43.089977 139851179493120 [compaction_job.cc:2437] [default] [JOB 30] Compacting 3@0 + 37@1 files to L1, score -1.00, subcompactions 3 : 3 2024/11/19-15:15:43.089996 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 27 Base level 0, inputs: [4284(85MB) 4276(515MB) 4269(517MB)], [4221(31MB) 4222(31MB) 4224(31MB) 4226(31MB) 4228(31MB) 4230(31MB) 4232(31MB) 4233(31MB) 4234(31MB) 4236(31MB) 4237(31MB) 4238(31MB) 4240(31MB) 4241(31MB) 4242(31MB) 4244(31MB) 4245(31MB) 4246(31MB) 4247(31MB) 4248(31MB) 4249(31MB) 4250(31MB) 4251(31MB) 4252(31MB) 4253(31MB) 4254(31MB) 4255(31MB) 4256(31MB) 4257(31MB) 4258(31MB) 4259(31MB) 4260(31MB) 4261(31MB) 4262(31MB) 4263(31MB) 4264(31MB) 4265(3810KB)] 2024/11/19-15:15:43.090559 139851179493120 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4351: 5 keys, 3698 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.090597 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000543090577, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4351, "file_size": 3698, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 103619499, "largest_seqno": 105612323, "table_properties": {"data_size": 576, "index_size": 2304, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 80, "raw_average_key_size": 16, "raw_value_size": 570, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4351, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.091084 139851179493120 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4354: 60 keys, 12275 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.091112 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000543091097, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4354, "file_size": 12275, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 102838073, "largest_seqno": 105611420, "table_properties": {"data_size": 7168, "index_size": 4288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1200, "raw_average_key_size": 20, "raw_value_size": 7144, "raw_average_value_size": 119, "num_data_blocks": 1, "num_entries": 60, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4354, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.118638 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4352: 78321 keys, 3587965 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.118711 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000543118682, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4352, "file_size": 3587965, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 104151008, "largest_seqno": 105631328, "table_properties": {"data_size": 1549888, "index_size": 2037248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1879704, "raw_average_key_size": 24, "raw_value_size": 1549857, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 78321, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4352, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.121678 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4353: 32070 keys, 33525440 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.121737 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543121717, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4353, "file_size": 33525440, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 99708797, "largest_seqno": 99770876, "table_properties": {"data_size": 32615424, "index_size": 909184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641400, "raw_average_key_size": 20, "raw_value_size": 32615422, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32070, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4353, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.128294 139851179493120 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4355: 49854 keys, 33506752 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.128334 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000543128314, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4355, "file_size": 33506752, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 102848077, "largest_seqno": 103546012, "table_properties": {"data_size": 32158912, "index_size": 1347008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196496, "raw_average_key_size": 24, "raw_value_size": 32158851, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49854, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4355, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.139041 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-00 2024/11/19-15:15:43.139133 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00029/att-00/compact.done) = No such file or directory: 0.685 sec on http probe failed: 5, giveup 2024/11/19-15:15:43.139151 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00029/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 6.230 sec, real 0.980 (submit 286.899 ms wait 0.685 s), input 6.230 MiB, send shutdown 2024/11/19-15:15:43.147028 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-00 2024/11/19-15:15:43.147221 139851127064320 [dcompact_etcd.cc:1773] job-00029/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:43.153137 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4357: 32151 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.153178 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543153158, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4357, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 99770877, "largest_seqno": 99833036, "table_properties": {"data_size": 32697600, "index_size": 826496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32697582, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4357, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.165942 139851179493120 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4358: 49876 keys, 33506688 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.165986 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000543165965, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4358, "file_size": 33506688, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 103378501, "largest_seqno": 104545024, "table_properties": {"data_size": 32156736, "index_size": 1349120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1197024, "raw_average_key_size": 24, "raw_value_size": 32156701, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49876, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4358, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.190882 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4359: 32059 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.190955 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543190916, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4359, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 99695093, "largest_seqno": 99835112, "table_properties": {"data_size": 32604160, "index_size": 920000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32604100, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4359, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.203318 139851179493120 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4360: 49870 keys, 33506496 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.203364 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000543203343, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4360, "file_size": 33506496, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 104137054, "largest_seqno": 105535453, "table_properties": {"data_size": 32158336, "index_size": 1347328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196880, "raw_average_key_size": 24, "raw_value_size": 32158296, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49870, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4360, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.226752 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4356: 231233 keys, 33325251 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.226805 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000543226784, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4356, "file_size": 33325251, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 102867083, "largest_seqno": 103447364, "table_properties": {"data_size": 27032832, "index_size": 6291584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6474524, "raw_average_key_size": 28, "raw_value_size": 27032774, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231233, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4356, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.227393 139851179493120 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4362: 30400 keys, 20479744 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.227437 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000543227417, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4362, "file_size": 20479744, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 104895917, "largest_seqno": 105615324, "table_properties": {"data_size": 19602624, "index_size": 876288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 729600, "raw_average_key_size": 24, "raw_value_size": 19602609, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 30400, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4362, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.227701 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4361: 32081 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.227738 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543227720, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4361, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 99745082, "largest_seqno": 99817175, "table_properties": {"data_size": 32628352, "index_size": 895488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641620, "raw_average_key_size": 20, "raw_value_size": 32628308, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32081, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4361, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.271534 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4365: 32057 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.271610 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543271578, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4365, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 99817176, "largest_seqno": 99868243, "table_properties": {"data_size": 32602624, "index_size": 921280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602598, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4365, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.290287 139851179493120 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4364: 180000 keys, 14209155 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.290346 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000543290323, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4364, "file_size": 14209155, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 102851078, "largest_seqno": 105618326, "table_properties": {"data_size": 9716672, "index_size": 4491648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3600000, "raw_average_key_size": 20, "raw_value_size": 9716646, "raw_average_value_size": 53, "num_data_blocks": 1, "num_entries": 180000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4364, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.310210 139851179493120 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4367: 53100 keys, 1446602 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.310264 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000543310242, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4367, "file_size": 1446602, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 102899964, "largest_seqno": 105612321, "table_properties": {"data_size": 53120, "index_size": 1392640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1274400, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 53100, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4367, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.320736 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4366: 32057 keys, 33524352 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.320801 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543320770, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4366, "file_size": 33524352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 99868244, "largest_seqno": 100699040, "table_properties": {"data_size": 32602560, "index_size": 920960, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602526, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4366, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.330742 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4363: 231476 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.330791 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000543330770, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4363, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 102915872, "largest_seqno": 103615594, "table_properties": {"data_size": 27026880, "index_size": 6297280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481328, "raw_average_key_size": 28, "raw_value_size": 27026844, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231476, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4363, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.345017 139851179493120 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4368: 101679 keys, 4618304 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.345067 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000543345044, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4368, "file_size": 4618304, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 102854079, "largest_seqno": 104554027, "table_properties": {"data_size": 2014144, "index_size": 2603328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2440296, "raw_average_key_size": 24, "raw_value_size": 2014143, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 101679, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4368, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.370115 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4369: 32112 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.370179 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543370149, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4369, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 100699041, "largest_seqno": 100761161, "table_properties": {"data_size": 32658432, "index_size": 865280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642240, "raw_average_key_size": 20, "raw_value_size": 32658395, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32112, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4369, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.407233 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4371: 32059 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.407296 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543407264, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4371, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 100761162, "largest_seqno": 100833232, "table_properties": {"data_size": 32602944, "index_size": 921152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32602903, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4371, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.417684 139851127064320 [dcompact_etcd.cc:1163] job-00029/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.270 sec, input = 6.230 MiB 2024/11/19-15:15:43.432391 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-02 2024/11/19-15:15:43.432489 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00025/att-02/compact.done) = No such file or directory: 0.561 sec on http probe failed: 1, retry 2024/11/19-15:15:43.432890 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4370: 231455 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.432950 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000543432916, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4370, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 103328072, "largest_seqno": 104109237, "table_properties": {"data_size": 27025280, "index_size": 6298880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480740, "raw_average_key_size": 28, "raw_value_size": 27025242, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231455, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4370, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.437996 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4372: 32059 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.438036 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543438016, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4372, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 100682861, "largest_seqno": 100842883, "table_properties": {"data_size": 32603456, "index_size": 920768, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603394, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4372, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.468128 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4374: 32150 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.468172 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543468151, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4374, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 100725275, "largest_seqno": 100787433, "table_properties": {"data_size": 32696512, "index_size": 827136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32696482, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4374, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.498473 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4375: 32059 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.498521 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543498500, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4375, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 100787434, "largest_seqno": 100852504, "table_properties": {"data_size": 32602688, "index_size": 921792, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32602651, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4375, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.528847 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4376: 32060 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.528898 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543528878, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4376, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 100852505, "largest_seqno": 101678763, "table_properties": {"data_size": 32603328, "index_size": 920384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32603269, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4376, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.535427 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4373: 231353 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.535478 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000543535457, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4373, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 103914423, "largest_seqno": 104615087, "table_properties": {"data_size": 27031104, "index_size": 6293120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6477884, "raw_average_key_size": 28, "raw_value_size": 27031082, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231353, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4373, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.559228 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4377: 32150 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.559282 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543559252, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4377, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 101678764, "largest_seqno": 101750926, "table_properties": {"data_size": 32697216, "index_size": 826432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32697160, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4377, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.583045 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-01 2024/11/19-15:15:43.583147 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00029/att-01/compact.done) = No such file or directory: 0.156 sec on http probe failed: 1, retry 2024/11/19-15:15:43.589309 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4379: 32057 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.589352 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543589332, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4379, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 101750927, "largest_seqno": 101812993, "table_properties": {"data_size": 32601536, "index_size": 922944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601522, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4379, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.620428 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4380: 32057 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.620477 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543620456, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4380, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 101680088, "largest_seqno": 101840113, "table_properties": {"data_size": 32601344, "index_size": 922816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601306, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4380, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.638479 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4378: 231405 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.638530 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000543638509, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4378, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 103938848, "largest_seqno": 104569473, "table_properties": {"data_size": 27029376, "index_size": 6294784, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6479340, "raw_average_key_size": 28, "raw_value_size": 27029370, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231405, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4378, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.650653 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4381: 32130 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.650698 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543650677, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4381, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 101705032, "largest_seqno": 101767172, "table_properties": {"data_size": 32678784, "index_size": 845248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642600, "raw_average_key_size": 20, "raw_value_size": 32678724, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32130, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4381, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.681489 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4383: 32055 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.681535 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543681515, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4383, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 101767173, "largest_seqno": 101829236, "table_properties": {"data_size": 32601984, "index_size": 921792, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32601958, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4383, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.711839 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4384: 32052 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.711885 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543711865, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4384, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 101829237, "largest_seqno": 102669221, "table_properties": {"data_size": 32596800, "index_size": 926976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641040, "raw_average_key_size": 20, "raw_value_size": 32596762, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32052, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4384, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.716875 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-01 2024/11/19-15:15:43.716979 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00029/att-01/compact.done) = No such file or directory: 0.290 sec on http probe failed: 2, retry 2024/11/19-15:15:43.742392 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4385: 32151 keys, 33525760 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.742440 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543742419, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4385, "file_size": 33525760, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 102669222, "largest_seqno": 102731383, "table_properties": {"data_size": 32697664, "index_size": 827264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32697609, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4385, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.742809 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4382: 231460 keys, 33324931 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.742851 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000543742831, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4382, "file_size": 33324931, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 104569474, "largest_seqno": 105319196, "table_properties": {"data_size": 27024832, "index_size": 6299264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480880, "raw_average_key_size": 28, "raw_value_size": 27024808, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231460, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4382, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.772831 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4386: 32053 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.772878 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543772857, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4386, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 102731384, "largest_seqno": 102793446, "table_properties": {"data_size": 32598592, "index_size": 925824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32598571, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4386, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.803370 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4388: 32057 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.803418 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543803397, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4388, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 102678045, "largest_seqno": 102838071, "table_properties": {"data_size": 32600128, "index_size": 924032, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32600109, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4388, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.833959 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4389: 32070 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.834006 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543833983, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4389, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 102685484, "largest_seqno": 102747564, "table_properties": {"data_size": 32613184, "index_size": 910976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641400, "raw_average_key_size": 20, "raw_value_size": 32613165, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32070, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4389, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.846818 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4387: 231454 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.846863 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000543846842, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4387, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 104913924, "largest_seqno": 105610517, "table_properties": {"data_size": 27025600, "index_size": 6298560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480712, "raw_average_key_size": 28, "raw_value_size": 27025576, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231454, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4387, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.850537 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-01 2024/11/19-15:15:43.850624 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00029/att-01/compact.done) = No such file or directory: 0.424 sec on http probe failed: 3, retry 2024/11/19-15:15:43.865330 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4390: 32055 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.865372 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543865353, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4390, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 102747565, "largest_seqno": 102819633, "table_properties": {"data_size": 32598720, "index_size": 925184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32598704, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4390, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.890000 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-02 2024/11/19-15:15:43.890086 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00025/att-02/compact.done) = No such file or directory: 1.019 sec on http probe failed: 2, retry 2024/11/19-15:15:43.908647 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4392: 32053 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.908696 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000543908674, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4392, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 102819634, "largest_seqno": 103629119, "table_properties": {"data_size": 32600768, "index_size": 923008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32600709, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4392, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.927890 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4391: 149782 keys, 21617283 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:43.927947 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000543927914, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4391, "file_size": 21617283, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 105182894, "largest_seqno": 105580598, "table_properties": {"data_size": 17493120, "index_size": 4123328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4193896, "raw_average_key_size": 28, "raw_value_size": 17493112, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 149782, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4391, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:43.984423 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-01 2024/11/19-15:15:43.984492 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00029/att-01/compact.done) = No such file or directory: 0.558 sec on http probe failed: 4, retry 2024/11/19-15:15:44.012661 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4393: 32085 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.012711 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000544012689, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4393, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 103629120, "largest_seqno": 103712266, "table_properties": {"data_size": 32630080, "index_size": 893632, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641700, "raw_average_key_size": 20, "raw_value_size": 32630077, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32085, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4393, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.012773 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4394: 32059 keys, 33524800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.012814 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000544012795, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4394, "file_size": 33524800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 96783529, "largest_seqno": 96845598, "table_properties": {"data_size": 32602624, "index_size": 921344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32602567, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000543, "oldest_key_time": 0, "file_creation_time": 1732000543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4394, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.071018 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4395: 32055 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.071074 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000544071052, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4395, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 103712267, "largest_seqno": 103774331, "table_properties": {"data_size": 32599872, "index_size": 923776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32599866, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4395, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.071315 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4396: 32059 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.071365 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000544071338, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4396, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 96691957, "largest_seqno": 96851981, "table_properties": {"data_size": 32603456, "index_size": 920960, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603394, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4396, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.075809 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-02 2024/11/19-15:15:44.075878 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00021/att-02/compact.done) = No such file or directory: 9.011 sec on http probe failed: 3, retry 2024/11/19-15:15:44.117883 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-01 2024/11/19-15:15:44.117966 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00029/att-01/compact.done) = No such file or directory: 0.691 sec on http probe failed: 5, giveup 2024/11/19-15:15:44.117983 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00029/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 6.230 sec, real 0.971 (submit 270.379 ms wait 0.691 s), input 6.230 MiB, send shutdown 2024/11/19-15:15:44.128056 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-01 2024/11/19-15:15:44.148071 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4397: 32150 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.148126 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000544148103, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4397, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 103774332, "largest_seqno": 103826489, "table_properties": {"data_size": 32696512, "index_size": 827136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32696481, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4397, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.148378 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4398: 32122 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.148425 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000544148402, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4398, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 96737639, "largest_seqno": 96799769, "table_properties": {"data_size": 32666624, "index_size": 857600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642440, "raw_average_key_size": 20, "raw_value_size": 32666608, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32122, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4398, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.148442 139851127064320 [dcompact_etcd.cc:1773] job-00029/att-01: CleanFiles 0.020 sec : 0.000 2024/11/19-15:15:44.304617 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4399: 32058 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.304672 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000544304650, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4399, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 103676560, "largest_seqno": 103826582, "table_properties": {"data_size": 32603840, "index_size": 919808, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603792, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4399, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.304718 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4400: 32058 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.304768 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000544304744, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4400, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 96799770, "largest_seqno": 96867843, "table_properties": {"data_size": 32603392, "index_size": 921024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603376, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4400, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.336664 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4402: 32058 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.336721 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000544336698, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4402, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 96867844, "largest_seqno": 97701739, "table_properties": {"data_size": 32603200, "index_size": 920704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603168, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4402, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.340200 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4401: 32150 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.340269 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000544340231, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4401, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 103738535, "largest_seqno": 103800694, "table_properties": {"data_size": 32697280, "index_size": 826560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32697278, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4401, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.347587 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-02 2024/11/19-15:15:44.347701 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00025/att-02/compact.done) = No such file or directory: 1.477 sec on http probe failed: 3, retry 2024/11/19-15:15:44.368133 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4403: 32150 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.368181 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000544368161, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4403, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 97701740, "largest_seqno": 97763898, "table_properties": {"data_size": 32697280, "index_size": 826624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32697266, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4403, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.396544 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4404: 32058 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.396596 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000544396574, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4404, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 103800695, "largest_seqno": 103861766, "table_properties": {"data_size": 32603072, "index_size": 921280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603021, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4404, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.407889 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4405: 32058 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.407942 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000544407916, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4405, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 97763899, "largest_seqno": 97825965, "table_properties": {"data_size": 32601856, "index_size": 921728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32601815, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4405, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.453896 139851127064320 [dcompact_etcd.cc:1163] job-00029/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.305 sec, input = 6.230 MiB 2024/11/19-15:15:44.464324 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4406: 32058 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.464376 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000544464353, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4406, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 103861767, "largest_seqno": 104694318, "table_properties": {"data_size": 32602944, "index_size": 920896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602935, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4406, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.464654 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4407: 32059 keys, 33524352 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.464699 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000544464679, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4407, "file_size": 33524352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 97689824, "largest_seqno": 97849847, "table_properties": {"data_size": 32603200, "index_size": 920320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603192, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4407, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.532230 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4409: 32061 keys, 33525376 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.532287 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000544532266, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4409, "file_size": 33525376, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 97718008, "largest_seqno": 97790080, "table_properties": {"data_size": 32605504, "index_size": 919040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641220, "raw_average_key_size": 20, "raw_value_size": 32605455, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32061, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4409, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.532373 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4408: 32116 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.532432 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000544532403, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4408, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 104694319, "largest_seqno": 104766446, "table_properties": {"data_size": 32662784, "index_size": 861376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642320, "raw_average_key_size": 20, "raw_value_size": 32662729, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32116, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4408, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.598188 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4410: 32055 keys, 33525376 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.598248 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000544598226, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4410, "file_size": 33525376, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 97790081, "largest_seqno": 97858149, "table_properties": {"data_size": 32601600, "index_size": 922944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32601546, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4410, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.598289 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4411: 32056 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.598335 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000544598315, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4411, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 104766447, "largest_seqno": 104828511, "table_properties": {"data_size": 32602560, "index_size": 921024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32602553, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4411, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.616190 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-02 2024/11/19-15:15:44.616297 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00029/att-02/compact.done) = No such file or directory: 0.156 sec on http probe failed: 1, retry 2024/11/19-15:15:44.660237 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4412: 32055 keys, 33524352 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.660299 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000544660277, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4412, "file_size": 33524352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 97858150, "largest_seqno": 98681367, "table_properties": {"data_size": 32599680, "index_size": 923840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32599653, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4412, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.660349 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4413: 32058 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.660410 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000544660379, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4413, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 104676086, "largest_seqno": 104836108, "table_properties": {"data_size": 32603712, "index_size": 920640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603707, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4413, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.706385 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4414: 32070 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.706444 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000544706421, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4414, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 98681368, "largest_seqno": 98743447, "table_properties": {"data_size": 32614336, "index_size": 909952, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641400, "raw_average_key_size": 20, "raw_value_size": 32614320, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32070, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4414, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.719685 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4415: 32152 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.719761 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000544719729, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4415, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 104720553, "largest_seqno": 104782713, "table_properties": {"data_size": 32697088, "index_size": 827008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643040, "raw_average_key_size": 20, "raw_value_size": 32697060, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32152, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4415, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.748946 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-02 2024/11/19-15:15:44.749048 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00029/att-02/compact.done) = No such file or directory: 0.287 sec on http probe failed: 2, retry 2024/11/19-15:15:44.755656 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4416: 32053 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.755709 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000544755686, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4416, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 98743448, "largest_seqno": 98805509, "table_properties": {"data_size": 32599040, "index_size": 925120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32598978, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4416, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.800822 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4417: 32058 keys, 33525376 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.800938 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000544800893, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4417, "file_size": 33525376, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 104782714, "largest_seqno": 104850784, "table_properties": {"data_size": 32602816, "index_size": 921728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602760, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4417, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.800983 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4418: 32056 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.801028 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000544801008, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4418, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 98687017, "largest_seqno": 98847040, "table_properties": {"data_size": 32600512, "index_size": 923328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32600456, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4418, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:44.805049 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-02 2024/11/19-15:15:44.805129 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00025/att-02/compact.done) = No such file or directory: 1.934 sec on http probe failed: 4, retry 2024/11/19-15:15:44.882003 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-02 2024/11/19-15:15:44.882088 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00029/att-02/compact.done) = No such file or directory: 0.420 sec on http probe failed: 3, retry 2024/11/19-15:15:44.933003 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4420: 32088 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:44.933058 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000544933035, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4420, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 98707551, "largest_seqno": 98769648, "table_properties": {"data_size": 32632384, "index_size": 891840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641760, "raw_average_key_size": 20, "raw_value_size": 32632333, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32088, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4420, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:45.011789 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-02 2024/11/19-15:15:45.011872 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00029/att-02/compact.done) = No such file or directory: 0.553 sec on http probe failed: 4, retry 2024/11/19-15:15:45.034737 139833664079616 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4419: 21329 keys, 22349184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:45.034789 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000545034767, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4419, "file_size": 22349184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 104850785, "largest_seqno": 105628327, "table_properties": {"data_size": 21689408, "index_size": 658944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 426580, "raw_average_key_size": 20, "raw_value_size": 21689397, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 21329, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4419, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:45.108895 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4421: 32056 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:45.108958 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000545108935, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4421, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 98769649, "largest_seqno": 98831713, "table_properties": {"data_size": 32599872, "index_size": 923968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32599857, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000544, "oldest_key_time": 0, "file_creation_time": 1732000544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4421, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:45.139988 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4422: 32055 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:45.140040 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000545140019, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4422, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 98831714, "largest_seqno": 99640799, "table_properties": {"data_size": 32601088, "index_size": 923200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32601030, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000545, "oldest_key_time": 0, "file_creation_time": 1732000545, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4422, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:45.145846 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-02 2024/11/19-15:15:45.145942 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00029/att-02/compact.done) = No such file or directory: 0.683 sec on http probe failed: 5, giveup 2024/11/19-15:15:45.145960 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00029/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 6.230 sec, real 0.997 (submit 305.358 ms wait 0.683 s), input 6.230 MiB, send shutdown 2024/11/19-15:15:45.155669 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00029/att-02 2024/11/19-15:15:45.155907 139851127064320 [dcompact_etcd.cc:1773] job-00029/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:45.157792 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00029) = success 2024/11/19-15:15:45.157801 139851127064320 [dcompact_etcd.cc:1773] job-00029/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:45.157820 139851127064320 (Original Log Time 2024/11/19-15:15:42.158463) EVENT_LOG_v1 {"time_micros": 1732000542158456, "job": 29, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4323], "score": 2.20068, "input_data_size": 5409904, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:45.157828 139851127064320 [compaction_job.cc:2437] [default] [JOB 29] Compacting 1@2 files to L3, score 2.20, subcompactions 1 : 1 2024/11/19-15:15:45.157835 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 27 Base level 2, inputs: [4323(5283KB)] 2024/11/19-15:15:45.163090 139851127064320 [compaction_job.cc:703] job-00029: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:15:45.163476 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:15:45.163569 139851127064320 (Original Log Time 2024/11/19-15:15:45.157866) EVENT_LOG_v1 {"time_micros": 1732000545157846, "job": 29, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4323], "score": 2.20068, "input_data_size": 5409904, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:45.163575 139851127064320 (Original Log Time 2024/11/19-15:15:45.163145) [compaction_job.cc:2022] [default] [JOB 29] Compacted 1@2 files to L3 => 0 bytes 2024/11/19-15:15:45.163578 139851127064320 (Original Log Time 2024/11/19-15:15:45.163449) [compaction_job.cc:1224] [default] compacted to: files[5 37 77 75 0 0 0] max score 2.20 (62 files need compaction), MB/sec: 1033.6 rd, 0.0 wr, level 3, files in(1, 0) out(0 +0 blob) MB in(5.2, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 45242, records dropped: 45242 output_compression: Snappy 2024/11/19-15:15:45.163585 139851127064320 (Original Log Time 2024/11/19-15:15:45.163469) EVENT_LOG_v1 {"time_micros": 1732000545163459, "cf": "default", "job": 29, "event": "compaction_finished", "compaction_time_micros": 5234, "compaction_time_cpu_micros": 5192, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 45242, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 37, 77, 75, 0, 0, 0]} 2024/11/19-15:15:45.163657 139851127064320 [compaction_job.cc:2437] [default] [JOB 31] Compacting 1@2 files to L3, score 2.20, subcompactions 1 : 1 2024/11/19-15:15:45.163664 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 29 Base level 2, inputs: [3809(31MB)] 2024/11/19-15:15:45.166159 139849937979136 [compaction_job.cc:1948] [default] [JOB 30] Generated table #4423: 26973 keys, 28166336 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:45.166198 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000545166179, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 4423, "file_size": 28166336, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 99640800, "largest_seqno": 99708796, "table_properties": {"data_size": 27430336, "index_size": 735168, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 539460, "raw_average_key_size": 20, "raw_value_size": 27430274, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 26973, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000545, "oldest_key_time": 0, "file_creation_time": 1732000545, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4423, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:15:45.166288 139851179493120 [compaction_job.cc:703] job-00030: subcompact[0], size: 0.141290 G, files: /storage/mytopling/.rocksdb [004351,004354,004355,004358,004360,004362,004364,004367,004368] 2024/11/19-15:15:45.166311 139851179493120 [compaction_job.cc:703] job-00030: subcompact[1], size: 0.856570 G, files: /storage/mytopling/.rocksdb [004352,004356,004363,004370,004373,004378,004382,004387,004391,004394,004396,004398,004400,004402,004403,004405,004407,004409,004410,004412,004414,004416,004418,004420,004421,004422,004423] 2024/11/19-15:15:45.166323 139851179493120 [compaction_job.cc:703] job-00030: subcompact[2], size: 1.229244 G, files: /storage/mytopling/.rocksdb [004353,004357,004359,004361,004365,004366,004369,004371,004372,004374,004375,004376,004377,004379,004380,004381,004383,004384,004385,004386,004388,004389,004390,004392,004393,004395,004397,004399,004401,004404,004406,004408,004411,004413,004415,004417,004419] 2024/11/19-15:15:45.246229 139851179493120 (Original Log Time 2024/11/19-15:15:43.090015) EVENT_LOG_v1 {"time_micros": 1732000543090006, "job": 30, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [4284, 4276, 4269], "files_L1": [4221, 4222, 4224, 4226, 4228, 4230, 4232, 4233, 4234, 4236, 4237, 4238, 4240, 4241, 4242, 4244, 4245, 4246, 4247, 4248, 4249, 4250, 4251, 4252, 4253, 4254, 4255, 4256, 4257, 4258, 4259, 4260, 4261, 4262, 4263, 4264, 4265], "score": -1, "input_data_size": 2382581835, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:45.246252 139851179493120 (Original Log Time 2024/11/19-15:15:45.245437) [compaction_job.cc:2022] [default] [JOB 30] Compacted 3@0 + 37@1 files to L1 => 2227103623 bytes 2024/11/19-15:15:45.246256 139851179493120 (Original Log Time 2024/11/19-15:15:45.245956) [compaction_job.cc:1224] [default] compacted to: files[2 73 77 75 0 0 0] max score 4.15 (134 files need compaction), MB/sec: 1147.5 rd, 1072.6 wr, level 1, files in(3, 37) out(73 +0 blob) MB in(1117.5, 1154.7 +0.0 blob) out(2123.9 +0.0 blob), read-write-amplify(3.9) write-amplify(1.9) OK, records in: 4111225, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:45.246260 139851179493120 (Original Log Time 2024/11/19-15:15:45.246005) EVENT_LOG_v1 {"time_micros": 1732000545245970, "cf": "default", "job": 30, "event": "compaction_finished", "compaction_time_micros": 2076311, "compaction_time_cpu_micros": 2931771, "output_level": 1, "num_output_files": 73, "total_output_size": 2227103623, "num_input_records": 4111225, "num_output_records": 4111225, "num_subcompactions": 3, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 73, 77, 75, 0, 0, 0]} 2024/11/19-15:15:45.246351 139834427442944 (Original Log Time 2024/11/19-15:14:42.257771) [db_impl_compaction_flush.cc:2043] [default] Manual compaction starting 2024/11/19-15:15:45.246473 139851179493120 [compaction_job.cc:2437] [default] [JOB 32] Compacting 1@1 files to L2, score 4.15, subcompactions 1 : 1 2024/11/19-15:15:45.246482 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 30 Base level 1, inputs: [4351(3698B)] 2024/11/19-15:15:45.258813 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-02 2024/11/19-15:15:45.258897 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00025/att-02/compact.done) = No such file or directory: 2.392 sec on http probe failed: 5, giveup 2024/11/19-15:15:45.258931 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00025/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 22.431 sec, real 2.684 (submit 287.845 ms wait 2.392 s), input 22.431 MiB, send shutdown 2024/11/19-15:15:45.263582 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00025/att-02 2024/11/19-15:15:45.263813 139850978166528 [dcompact_etcd.cc:1773] job-00025/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:45.265616 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00025) = success 2024/11/19-15:15:45.265628 139850978166528 [dcompact_etcd.cc:1773] job-00025/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:45.265649 139850978166528 (Original Log Time 2024/11/19-15:15:37.225188) EVENT_LOG_v1 {"time_micros": 1732000537225183, "job": 25, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4292], "score": 2.3634, "input_data_size": 13680446, "oldest_snapshot_seqno": 108863688} 2024/11/19-15:15:45.265658 139850978166528 [compaction_job.cc:2437] [default] [JOB 25] Compacting 1@2 files to L3, score 2.36, subcompactions 1 : 1 2024/11/19-15:15:45.265667 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 23 Base level 2, inputs: [4292(13MB)] 2024/11/19-15:15:45.346577 139850978166528 [compaction_job.cc:703] job-00025: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:15:45.347048 139850978166528 (Original Log Time 2024/11/19-15:15:45.265696) EVENT_LOG_v1 {"time_micros": 1732000545265678, "job": 25, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4292], "score": 2.3634, "input_data_size": 13680446, "oldest_snapshot_seqno": 108863688} 2024/11/19-15:15:45.347057 139850978166528 (Original Log Time 2024/11/19-15:15:45.346634) [compaction_job.cc:2022] [default] [JOB 25] Compacted 1@2 files to L3 => 0 bytes 2024/11/19-15:15:45.347061 139850978166528 (Original Log Time 2024/11/19-15:15:45.346966) [compaction_job.cc:1224] [default] compacted to: files[2 73 76 75 0 0 0] max score 4.15 (132 files need compaction), MB/sec: 169.1 rd, 0.0 wr, level 3, files in(1, 0) out(0 +0 blob) MB in(13.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 537000, records dropped: 537000 output_compression: Snappy 2024/11/19-15:15:45.347077 139850978166528 (Original Log Time 2024/11/19-15:15:45.346988) EVENT_LOG_v1 {"time_micros": 1732000545346976, "cf": "default", "job": 25, "event": "compaction_finished", "compaction_time_micros": 80896, "compaction_time_cpu_micros": 80628, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 537000, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 73, 76, 75, 0, 0, 0]} 2024/11/19-15:15:45.347110 139850978166528 (Original Log Time 2024/11/19-15:15:45.347092) [db_impl_compaction_flush.cc:3374] [default] Manual compaction from level-1 to level-2 from '00000131' seq:72057594037927935, type:22 .. '00000132' seq:0, type:0; will stop at (end) 2024/11/19-15:15:45.347115 139850978166528 [compaction_job.cc:2437] [default] [JOB 33] Compacting 1@1 files to L2, score -1.00, subcompactions 1 : 1 2024/11/19-15:15:45.347121 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 30 Base level 1, inputs: [4367(1412KB)] 2024/11/19-15:15:45.456793 139851127064320 [dcompact_etcd.cc:1163] job-00031/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.293 sec, input = 31.702 MiB 2024/11/19-15:15:45.539720 139851179493120 [dcompact_etcd.cc:1163] job-00032/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.293 sec, input = 650 B 2024/11/19-15:15:45.554742 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-00 2024/11/19-15:15:45.554806 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00032/att-00/compact.done) = No such file or directory: 0.006 sec on http probe failed: 1, retry 2024/11/19-15:15:45.563872 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-00 2024/11/19-15:15:45.563929 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00032/att-00/compact.done) = No such file or directory: 0.015 sec on http probe failed: 2, retry 2024/11/19-15:15:45.574155 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-00 2024/11/19-15:15:45.574203 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00032/att-00/compact.done) = No such file or directory: 0.024 sec on http probe failed: 3, retry 2024/11/19-15:15:45.578396 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-00 2024/11/19-15:15:45.578440 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00032/att-00/compact.done) = No such file or directory: 0.035 sec on http probe failed: 4, retry 2024/11/19-15:15:45.583002 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-00 2024/11/19-15:15:45.583049 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00032/att-00/compact.done) = No such file or directory: 0.039 sec on http probe failed: 5, giveup 2024/11/19-15:15:45.583063 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00032/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.336 (submit 292.728 ms wait 0.039 s), input 650 B, send shutdown 2024/11/19-15:15:45.588284 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-00 2024/11/19-15:15:45.588491 139851179493120 [dcompact_etcd.cc:1773] job-00032/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:45.643020 139850978166528 [dcompact_etcd.cc:1163] job-00033/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.295 sec, input = 1.215 MiB 2024/11/19-15:15:45.682606 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-00 2024/11/19-15:15:45.682675 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00033/att-00/compact.done) = No such file or directory: 0.030 sec on http probe failed: 1, retry 2024/11/19-15:15:45.712776 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-00 2024/11/19-15:15:45.712824 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00033/att-00/compact.done) = No such file or directory: 0.064 sec on http probe failed: 2, retry 2024/11/19-15:15:45.742066 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-00 2024/11/19-15:15:45.742111 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00033/att-00/compact.done) = No such file or directory: 0.094 sec on http probe failed: 3, retry 2024/11/19-15:15:45.774250 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-00 2024/11/19-15:15:45.774301 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00033/att-00/compact.done) = No such file or directory: 0.123 sec on http probe failed: 4, retry 2024/11/19-15:15:45.803597 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-00 2024/11/19-15:15:45.803660 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00033/att-00/compact.done) = No such file or directory: 0.156 sec on http probe failed: 5, giveup 2024/11/19-15:15:45.803674 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00033/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.456 (submit 295.495 ms wait 0.156 s), input 1.215 MiB, send shutdown 2024/11/19-15:15:45.808335 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-00 2024/11/19-15:15:45.808547 139850978166528 [dcompact_etcd.cc:1773] job-00033/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:45.850315 139851179493120 [dcompact_etcd.cc:1163] job-00032/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.262 sec, input = 650 B 2024/11/19-15:15:45.863425 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-01 2024/11/19-15:15:45.863496 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00032/att-01/compact.done) = No such file or directory: 0.006 sec on http probe failed: 1, retry 2024/11/19-15:15:45.872298 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-01 2024/11/19-15:15:45.872348 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00032/att-01/compact.done) = No such file or directory: 0.013 sec on http probe failed: 2, retry 2024/11/19-15:15:45.877359 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-01 2024/11/19-15:15:45.877405 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00032/att-01/compact.done) = No such file or directory: 0.022 sec on http probe failed: 3, retry 2024/11/19-15:15:45.885909 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-01 2024/11/19-15:15:45.885960 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00032/att-01/compact.done) = No such file or directory: 0.027 sec on http probe failed: 4, retry 2024/11/19-15:15:45.891194 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-01 2024/11/19-15:15:45.891240 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00032/att-01/compact.done) = No such file or directory: 0.036 sec on http probe failed: 5, giveup 2024/11/19-15:15:45.891254 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00032/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.303 (submit 261.724 ms wait 0.036 s), input 650 B, send shutdown 2024/11/19-15:15:45.898572 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-01 2024/11/19-15:15:45.898772 139851179493120 [dcompact_etcd.cc:1773] job-00032/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:46.061686 139850978166528 [dcompact_etcd.cc:1163] job-00033/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.253 sec, input = 1.215 MiB 2024/11/19-15:15:46.080676 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-02 2024/11/19-15:15:46.080741 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00021/att-02/compact.done) = No such file or directory: 11.017 sec on http probe failed: 4, retry 2024/11/19-15:15:46.100856 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-01 2024/11/19-15:15:46.100928 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00033/att-01/compact.done) = No such file or directory: 0.030 sec on http probe failed: 1, retry 2024/11/19-15:15:46.134823 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-01 2024/11/19-15:15:46.134867 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00033/att-01/compact.done) = No such file or directory: 0.064 sec on http probe failed: 2, retry 2024/11/19-15:15:46.161595 139851179493120 [dcompact_etcd.cc:1163] job-00032/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.263 sec, input = 650 B 2024/11/19-15:15:46.164663 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-01 2024/11/19-15:15:46.164709 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00033/att-01/compact.done) = No such file or directory: 0.098 sec on http probe failed: 3, retry 2024/11/19-15:15:46.168018 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-02 2024/11/19-15:15:46.168075 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00032/att-02/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:15:46.176701 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-02 2024/11/19-15:15:46.176746 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00032/att-02/compact.done) = No such file or directory: 0.007 sec on http probe failed: 2, retry 2024/11/19-15:15:46.184678 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-02 2024/11/19-15:15:46.184722 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00032/att-02/compact.done) = No such file or directory: 0.015 sec on http probe failed: 3, retry 2024/11/19-15:15:46.189240 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-02 2024/11/19-15:15:46.189285 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00032/att-02/compact.done) = No such file or directory: 0.023 sec on http probe failed: 4, retry 2024/11/19-15:15:46.196655 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-02 2024/11/19-15:15:46.196678 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-01 2024/11/19-15:15:46.196698 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00032/att-02/compact.done) = No such file or directory: 0.028 sec on http probe failed: 5, giveup 2024/11/19-15:15:46.196711 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00032/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.298 (submit 262.731 ms wait 0.028 s), input 650 B, send shutdown 2024/11/19-15:15:46.196735 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00033/att-01/compact.done) = No such file or directory: 0.127 sec on http probe failed: 4, retry 2024/11/19-15:15:46.204997 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00032/att-02 2024/11/19-15:15:46.205198 139851179493120 [dcompact_etcd.cc:1773] job-00032/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:46.207044 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00032) = success 2024/11/19-15:15:46.207052 139851179493120 [dcompact_etcd.cc:1773] job-00032/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:46.207070 139851179493120 (Original Log Time 2024/11/19-15:15:45.246495) EVENT_LOG_v1 {"time_micros": 1732000545246489, "job": 32, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4351], "score": 4.15081, "input_data_size": 3698, "oldest_snapshot_seqno": 109294689} 2024/11/19-15:15:46.207077 139851179493120 [compaction_job.cc:2437] [default] [JOB 32] Compacting 1@1 files to L2, score 4.15, subcompactions 1 : 1 2024/11/19-15:15:46.207083 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 30 Base level 1, inputs: [4351(3698B)] 2024/11/19-15:15:46.207315 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004424.sst) /storage/mytopling/..//tmp/Topling-82038-canJku 2024/11/19-15:15:46.207409 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004424.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:15:46.207489 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004424.sst): index pass time = 0.00's, 0.311'MB/sec index type = UintIndex_AllOne usrkeys = 5 min-keylen = 4 max-keylen = 4 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 32.00 2024/11/19-15:15:46.207564 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004424.sst): first pass time = 0.00's, 2.704'MB/sec 2024/11/19-15:15:46.207591 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004424.sst): old prealloc_size = 4067, real_size = 1296 2024/11/19-15:15:46.207606 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004424.sst): index type = UintIndex_AllOne , store type = MixedLenBlobStore usrkeys = 5 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 32.00 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 114.00 avg-zval = 185.60 2024/11/19-15:15:46.207743 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004424.sst): second pass time = 0.00's,7402.597'MB/sec, value only(87.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 15.988'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 11.577'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.137'MB/sec write SST data time = 0.00's, 19.455'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 7402.597'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 5 avg-key = 16.00 avg-zkey = 38.40 avg-val = 114.00 avg-zval = 185.60 usrkeys = 5 avg-key = 8.00 avg-zkey = 38.40 avg-val = 114.00 avg-zval = 185.60 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 4 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.2083 value = 0.6142 dict = 0.00 all = 0.2931 } Zip/UnZip{ index = 4.8000 value = 1.6281 dict = 0.00 all = 3.4123 } ---------------------------- total value len = 1.472612 GB avg = 0.542 KB (by entry num) total key len = 0.065824 GB avg = 0.024 KB total ukey len = 0.044084 GB avg = 0.016 KB total ukey num = 0.002717520 Billion total entry num = 0.002717520 Billion write speed all = 29.885840121 MB/sec (with seq num) write speed all = 29.463513360 MB/sec (without seq num) 2024/11/19-15:15:46.207912 139851179493120 [compaction_job.cc:1948] [default] [JOB 32] Generated table #4424: 5 keys, 2218 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:46.207959 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000546207937, "cf_name": "default", "job": 32, "event": "table_file_creation", "file_number": 4424, "file_size": 2218, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 928, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 80, "raw_average_key_size": 16, "raw_value_size": 570, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-MixedLenBlobStore", "creation_time": 1732000546, "oldest_key_time": 0, "file_creation_time": 1732000546, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4424, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:46.208042 139851179493120 [compaction_job.cc:703] job-00032: subcompact[0], size: 0.000002 G, files: /storage/mytopling/.rocksdb [004424] 2024/11/19-15:15:46.208604 139851179493120 (Original Log Time 2024/11/19-15:15:46.207108) EVENT_LOG_v1 {"time_micros": 1732000546207094, "job": 32, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4351], "score": 4.15081, "input_data_size": 3698, "oldest_snapshot_seqno": 109294689} 2024/11/19-15:15:46.208612 139851179493120 (Original Log Time 2024/11/19-15:15:46.208215) [compaction_job.cc:2022] [default] [JOB 32] Compacted 1@1 files to L2 => 2218 bytes 2024/11/19-15:15:46.208616 139851179493120 (Original Log Time 2024/11/19-15:15:46.208525) [compaction_job.cc:1224] [default] compacted to: files[2 72 77 75 0 0 0] max score 4.15 (133 files need compaction), MB/sec: 3.9 rd, 2.4 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.6) write-amplify(0.6) OK, records in: 5, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:46.208621 139851179493120 (Original Log Time 2024/11/19-15:15:46.208541) EVENT_LOG_v1 {"time_micros": 1732000546208533, "cf": "default", "job": 32, "event": "compaction_finished", "compaction_time_micros": 938, "compaction_time_cpu_micros": 832, "output_level": 2, "num_output_files": 1, "total_output_size": 2218, "num_input_records": 5, "num_output_records": 5, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 72, 77, 75, 0, 0, 0]} 2024/11/19-15:15:46.208689 139851179493120 [compaction_job.cc:2437] [default] [JOB 34] Compacting 1@1 files to L2, score 4.15, subcompactions 1 : 1 2024/11/19-15:15:46.208696 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 32 Base level 1, inputs: [4354(11KB)] 2024/11/19-15:15:46.230082 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-01 2024/11/19-15:15:46.230129 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00033/att-01/compact.done) = No such file or directory: 0.159 sec on http probe failed: 5, giveup 2024/11/19-15:15:46.230142 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00033/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.422 (submit 253.065 ms wait 0.159 s), input 1.215 MiB, send shutdown 2024/11/19-15:15:46.237410 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-01 2024/11/19-15:15:46.237585 139850978166528 [dcompact_etcd.cc:1773] job-00033/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:46.257872 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-00 2024/11/19-15:15:46.257950 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00031/att-00/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:15:46.517662 139851179493120 [dcompact_etcd.cc:1163] job-00034/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.309 sec, input = 8.148 KiB 2024/11/19-15:15:46.525437 139850978166528 [dcompact_etcd.cc:1163] job-00033/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.288 sec, input = 1.215 MiB 2024/11/19-15:15:46.526367 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-00 2024/11/19-15:15:46.526432 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00034/att-00/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:15:46.535888 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-00 2024/11/19-15:15:46.535942 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00034/att-00/compact.done) = No such file or directory: 0.009 sec on http probe failed: 2, retry 2024/11/19-15:15:46.544019 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-00 2024/11/19-15:15:46.544065 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00034/att-00/compact.done) = No such file or directory: 0.019 sec on http probe failed: 3, retry 2024/11/19-15:15:46.551770 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-00 2024/11/19-15:15:46.551814 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00034/att-00/compact.done) = No such file or directory: 0.027 sec on http probe failed: 4, retry 2024/11/19-15:15:46.556567 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-00 2024/11/19-15:15:46.556616 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00034/att-00/compact.done) = No such file or directory: 0.034 sec on http probe failed: 5, giveup 2024/11/19-15:15:46.556630 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00034/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.348 (submit 308.539 ms wait 0.034 s), input 8.148 KiB, send shutdown 2024/11/19-15:15:46.564547 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-02 2024/11/19-15:15:46.564611 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00033/att-02/compact.done) = No such file or directory: 0.030 sec on http probe failed: 1, retry 2024/11/19-15:15:46.564959 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-00 2024/11/19-15:15:46.565168 139851179493120 [dcompact_etcd.cc:1773] job-00034/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:46.594042 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-02 2024/11/19-15:15:46.594122 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00033/att-02/compact.done) = No such file or directory: 0.064 sec on http probe failed: 2, retry 2024/11/19-15:15:46.626700 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-02 2024/11/19-15:15:46.626784 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00033/att-02/compact.done) = No such file or directory: 0.093 sec on http probe failed: 3, retry 2024/11/19-15:15:46.656734 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-02 2024/11/19-15:15:46.656794 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00033/att-02/compact.done) = No such file or directory: 0.126 sec on http probe failed: 4, retry 2024/11/19-15:15:46.690259 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-02 2024/11/19-15:15:46.690311 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00033/att-02/compact.done) = No such file or directory: 0.156 sec on http probe failed: 5, giveup 2024/11/19-15:15:46.690326 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00033/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.453 (submit 287.796 ms wait 0.156 s), input 1.215 MiB, send shutdown 2024/11/19-15:15:46.694825 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00033/att-02 2024/11/19-15:15:46.708462 139850978166528 [dcompact_etcd.cc:1773] job-00033/att-02: CleanFiles 0.014 sec : 0.000 2024/11/19-15:15:46.719005 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00033) = success 2024/11/19-15:15:46.719017 139850978166528 [dcompact_etcd.cc:1773] job-00033/att-03: CleanFiles 0.011 sec : 0.000 2024/11/19-15:15:46.719037 139850978166528 (Original Log Time 2024/11/19-15:15:45.347132) EVENT_LOG_v1 {"time_micros": 1732000545347126, "job": 33, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L1": [4367], "score": -1, "input_data_size": 1446602, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:46.719044 139850978166528 [compaction_job.cc:2437] [default] [JOB 33] Compacting 1@1 files to L2, score -1.00, subcompactions 1 : 1 2024/11/19-15:15:46.719051 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 30 Base level 1, inputs: [4367(1412KB)] 2024/11/19-15:15:46.719269 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004425.sst) /storage/mytopling/..//tmp/Topling-82038-0lg8kn 2024/11/19-15:15:46.731825 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004425.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0005 GB, waited 0.000 sec, Key+Value = 0.001 GB 2024/11/19-15:15:46.732495 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004425.sst): first pass time = 0.01's, 96.436'MB/sec 2024/11/19-15:15:46.734792 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004425.sst): index pass time = 0.00's, 216.394'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 53100 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0006 GB zip-key = 0.0001 GB avg-key = 12.00 avg-zkey = 1.72 2024/11/19-15:15:46.734860 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004425.sst): old prealloc_size = 1591262, real_size = 104672 2024/11/19-15:15:46.734904 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004425.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZeroLengthBlobStore usrkeys = 53100 prefix = 4 raw-key = 0.0006 GB zip-key = 0.0001 GB avg-key = 12.00 avg-zkey = 1.72 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 0.00 avg-zval = 0.00 2024/11/19-15:15:46.734950 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004425.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.003 sec, Key+Value = 0.001 GB 2024/11/19-15:15:46.738747 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004425.sst): second pass time = 0.00's, 0.000'MB/sec, value only( 0.0% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 23.906'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2202.683'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 125.732'MB/sec write SST data time = 0.00's, 24.840'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 0.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 53100 avg-key = 24.00 avg-zkey = 1.72 avg-val = 0.00 avg-zval = 0.00 usrkeys = 53100 avg-key = 16.00 avg-zkey = 1.72 avg-val = 0.00 avg-zval = 0.00 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 5 UnZipSize{ index = 0.0008 GB value = 0.0000 GB dict = 0.00 MB all = 0.0013 GB } __ZipSize{ index = 0.0001 GB value = 0.0000 GB dict = 0.00 MB all = 0.0001 GB } UnZip/Zip{ index = 9.3223 value = 0.0000 dict = 0.00 all = 13.7338 } Zip/UnZip{ index = 0.1073 value = inf dict = 0.00 all = 0.0728 } ---------------------------- total value len = 1.472612 GB avg = 0.532 KB (by entry num) total key len = 0.067099 GB avg = 0.024 KB total ukey len = 0.044934 GB avg = 0.016 KB total ukey num = 0.002770620 Billion total entry num = 0.002770620 Billion write speed all = 29.605210040 MB/sec (with seq num) write speed all = 29.179027272 MB/sec (without seq num) 2024/11/19-15:15:46.739002 139850978166528 [compaction_job.cc:1948] [default] [JOB 33] Generated table #4425: 53100 keys, 92793 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:46.739038 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000546739018, "cf_name": "default", "job": 33, "event": "table_file_creation", "file_number": 4425, "file_size": 92793, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 80, "index_size": 91136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1274400, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 53100, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZeroLengthBlobStore", "creation_time": 1732000546, "oldest_key_time": 0, "file_creation_time": 1732000546, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4425, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:46.739268 139850978166528 [compaction_job.cc:703] job-00033: subcompact[0], size: 0.000093 G, files: /storage/mytopling/.rocksdb [004425] 2024/11/19-15:15:46.739908 139850978166528 (Original Log Time 2024/11/19-15:15:46.719082) EVENT_LOG_v1 {"time_micros": 1732000546719063, "job": 33, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L1": [4367], "score": -1, "input_data_size": 1446602, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:46.739917 139850978166528 (Original Log Time 2024/11/19-15:15:46.739498) [compaction_job.cc:2022] [default] [JOB 33] Compacted 1@1 files to L2 => 92793 bytes 2024/11/19-15:15:46.739934 139850978166528 (Original Log Time 2024/11/19-15:15:46.739827) [compaction_job.cc:1224] [default] compacted to: files[2 71 78 75 0 0 0] max score 4.15 (133 files need compaction), MB/sec: 71.6 rd, 4.6 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(1.4, 0.0 +0.0 blob) out(0.1 +0.0 blob), read-write-amplify(1.1) write-amplify(0.1) OK, records in: 53100, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:46.739942 139850978166528 (Original Log Time 2024/11/19-15:15:46.739846) EVENT_LOG_v1 {"time_micros": 1732000546739837, "cf": "default", "job": 33, "event": "compaction_finished", "compaction_time_micros": 20191, "compaction_time_cpu_micros": 17681, "output_level": 2, "num_output_files": 1, "total_output_size": 92793, "num_input_records": 53100, "num_output_records": 53100, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 71, 78, 75, 0, 0, 0]} 2024/11/19-15:15:46.740026 139834427442944 (Original Log Time 2024/11/19-15:15:45.246387) [db_impl_compaction_flush.cc:2043] [default] Manual compaction starting 2024/11/19-15:15:46.740130 139850978166528 [compaction_job.cc:2437] [default] [JOB 35] Compacting 4@1 files to L2, score 4.15, subcompactions 1 : 1 2024/11/19-15:15:46.740138 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 33 Base level 1, inputs: [4355(31MB) 4358(31MB) 4360(31MB) 4362(19MB)] 2024/11/19-15:15:46.900817 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-00 2024/11/19-15:15:46.900900 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00031/att-00/compact.done) = No such file or directory: 1.435 sec on http probe failed: 2, retry 2024/11/19-15:15:46.921500 139851179493120 [dcompact_etcd.cc:1163] job-00034/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.356 sec, input = 8.148 KiB 2024/11/19-15:15:46.929338 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-01 2024/11/19-15:15:46.929398 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00034/att-01/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:15:46.937018 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-01 2024/11/19-15:15:46.937063 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00034/att-01/compact.done) = No such file or directory: 0.008 sec on http probe failed: 2, retry 2024/11/19-15:15:46.944611 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-01 2024/11/19-15:15:46.944654 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00034/att-01/compact.done) = No such file or directory: 0.016 sec on http probe failed: 3, retry 2024/11/19-15:15:46.950242 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-01 2024/11/19-15:15:46.950283 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00034/att-01/compact.done) = No such file or directory: 0.023 sec on http probe failed: 4, retry 2024/11/19-15:15:46.959211 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-01 2024/11/19-15:15:46.959257 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00034/att-01/compact.done) = No such file or directory: 0.029 sec on http probe failed: 5, giveup 2024/11/19-15:15:46.959270 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00034/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.394 (submit 356.223 ms wait 0.029 s), input 8.148 KiB, send shutdown 2024/11/19-15:15:46.967527 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-01 2024/11/19-15:15:46.967728 139851179493120 [dcompact_etcd.cc:1773] job-00034/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:47.020313 139850978166528 [dcompact_etcd.cc:1163] job-00035/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 114.819 MiB 2024/11/19-15:15:47.241651 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-00 2024/11/19-15:15:47.241762 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00026/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:15:47.256890 139851179493120 [dcompact_etcd.cc:1163] job-00034/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 8.148 KiB 2024/11/19-15:15:47.277146 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-02 2024/11/19-15:15:47.277212 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00034/att-02/compact.done) = No such file or directory: 0.013 sec on http probe failed: 1, retry 2024/11/19-15:15:47.285964 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-02 2024/11/19-15:15:47.286012 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00034/att-02/compact.done) = No such file or directory: 0.021 sec on http probe failed: 2, retry 2024/11/19-15:15:47.294212 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-02 2024/11/19-15:15:47.294258 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00034/att-02/compact.done) = No such file or directory: 0.029 sec on http probe failed: 3, retry 2024/11/19-15:15:47.302645 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-02 2024/11/19-15:15:47.302690 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00034/att-02/compact.done) = No such file or directory: 0.038 sec on http probe failed: 4, retry 2024/11/19-15:15:47.312601 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-02 2024/11/19-15:15:47.312651 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00034/att-02/compact.done) = No such file or directory: 0.046 sec on http probe failed: 5, giveup 2024/11/19-15:15:47.312666 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00034/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.345 (submit 289.067 ms wait 0.046 s), input 8.148 KiB, send shutdown 2024/11/19-15:15:47.319888 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00034/att-02 2024/11/19-15:15:47.320100 139851179493120 [dcompact_etcd.cc:1773] job-00034/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:47.321869 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00034) = success 2024/11/19-15:15:47.321877 139851179493120 [dcompact_etcd.cc:1773] job-00034/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:47.321896 139851179493120 (Original Log Time 2024/11/19-15:15:46.208706) EVENT_LOG_v1 {"time_micros": 1732000546208701, "job": 34, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4354], "score": 4.14811, "input_data_size": 12275, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:47.321904 139851179493120 [compaction_job.cc:2437] [default] [JOB 34] Compacting 1@1 files to L2, score 4.15, subcompactions 1 : 1 2024/11/19-15:15:47.321910 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 32 Base level 1, inputs: [4354(11KB)] 2024/11/19-15:15:47.322175 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004426.sst) /storage/mytopling/..//tmp/Topling-82038-rFYRwB 2024/11/19-15:15:47.322305 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004426.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:15:47.322460 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004426.sst): first pass time = 0.00's, 30.368'MB/sec 2024/11/19-15:15:47.322513 139834066732800 [fixed_len_key_index.cc:363] FixedLenKeyIndex build_cache Finish: num_keys = 60, fixlen = 8, pref_len = 4, cplen = 3, time = 0.000 sec 2024/11/19-15:15:47.322573 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004426.sst): index pass time = 0.00's, 1.913'MB/sec index type = FixedLenKeyIndex usrkeys = 60 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 162.13 2024/11/19-15:15:47.322615 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004426.sst): old prealloc_size = 13502, real_size = 17392 2024/11/19-15:15:47.322631 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004426.sst): index type = FixedLenKeyIndex , store type = PlainBlobStore usrkeys = 60 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 162.13 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 119.07 avg-zval = 124.27 2024/11/19-15:15:47.322788 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004426.sst): second pass time = 0.00's,41534.884'MB/sec, value only(85.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 112.379'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 631.115'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.951'MB/sec write SST data time = 0.00's, 137.978'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 41534.884'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 60 avg-key = 20.00 avg-zkey = 162.13 avg-val = 119.07 avg-zval = 124.27 usrkeys = 60 avg-key = 12.00 avg-zkey = 162.13 avg-val = 119.07 avg-zval = 124.27 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 3 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.0740 value = 0.9582 dict = 0.00 all = 0.4564 } Zip/UnZip{ index = 13.5111 value = 1.0437 dict = 0.00 all = 2.1910 } ---------------------------- total value len = 1.472619 GB avg = 0.532 KB (by entry num) total key len = 0.067100 GB avg = 0.024 KB total ukey len = 0.044934 GB avg = 0.016 KB total ukey num = 0.002770680 Billion total entry num = 0.002770680 Billion write speed all = 29.276599394 MB/sec (with seq num) write speed all = 28.855140307 MB/sec (without seq num) 2024/11/19-15:15:47.322977 139851179493120 [compaction_job.cc:1948] [default] [JOB 34] Generated table #4426: 60 keys, 18282 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:47.323013 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000547322992, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 4426, "file_size": 18282, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 7456, "index_size": 9728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1200, "raw_average_key_size": 20, "raw_value_size": 7144, "raw_average_value_size": 119, "num_data_blocks": 1, "num_entries": 60, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-FixedLenKeyIndex-PlainBlobStore", "creation_time": 1732000547, "oldest_key_time": 0, "file_creation_time": 1732000547, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4426, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:47.323105 139851179493120 [compaction_job.cc:703] job-00034: subcompact[0], size: 0.000018 G, files: /storage/mytopling/.rocksdb [004426] 2024/11/19-15:15:47.323709 139851179493120 (Original Log Time 2024/11/19-15:15:47.321948) EVENT_LOG_v1 {"time_micros": 1732000547321931, "job": 34, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4354], "score": 4.14811, "input_data_size": 12275, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:47.323718 139851179493120 (Original Log Time 2024/11/19-15:15:47.323310) [compaction_job.cc:2022] [default] [JOB 34] Compacted 1@1 files to L2 => 18282 bytes 2024/11/19-15:15:47.323722 139851179493120 (Original Log Time 2024/11/19-15:15:47.323624) [compaction_job.cc:1224] [default] compacted to: files[2 70 79 75 0 0 0] max score 3.92 (129 files need compaction), MB/sec: 10.6 rd, 15.7 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(2.5) write-amplify(1.5) OK, records in: 60, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:47.323737 139851179493120 (Original Log Time 2024/11/19-15:15:47.323642) EVENT_LOG_v1 {"time_micros": 1732000547323632, "cf": "default", "job": 34, "event": "compaction_finished", "compaction_time_micros": 1162, "compaction_time_cpu_micros": 941, "output_level": 2, "num_output_files": 1, "total_output_size": 18282, "num_input_records": 60, "num_output_records": 60, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 70, 79, 75, 0, 0, 0]} 2024/11/19-15:15:47.323768 139851179493120 (Original Log Time 2024/11/19-15:15:47.323751) [db_impl_compaction_flush.cc:3374] [default] Manual compaction from level-2 to level-3 from '00000131' seq:72057594037927935, type:22 .. '00000132' seq:0, type:0; will stop at (end) 2024/11/19-15:15:47.323773 139851179493120 [compaction_job.cc:2437] [default] [JOB 36] Compacting 1@2 files to L3, score -1.00, subcompactions 1 : 1 2024/11/19-15:15:47.323779 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 33 Base level 2, inputs: [4425(90KB)] 2024/11/19-15:15:47.382880 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-00 2024/11/19-15:15:47.382983 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00027/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:15:47.416686 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-00 2024/11/19-15:15:47.416764 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00028/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:15:47.539672 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-00 2024/11/19-15:15:47.539748 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00031/att-00/compact.done) = No such file or directory: 2.078 sec on http probe failed: 3, retry 2024/11/19-15:15:47.618809 139851179493120 [dcompact_etcd.cc:1163] job-00036/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.295 sec, input = 1.215 MiB 2024/11/19-15:15:47.657815 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-00 2024/11/19-15:15:47.657900 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00036/att-00/compact.done) = No such file or directory: 0.030 sec on http probe failed: 1, retry 2024/11/19-15:15:47.691779 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-00 2024/11/19-15:15:47.691858 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00036/att-00/compact.done) = No such file or directory: 0.063 sec on http probe failed: 2, retry 2024/11/19-15:15:47.724935 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-00 2024/11/19-15:15:47.724996 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00036/att-00/compact.done) = No such file or directory: 0.097 sec on http probe failed: 3, retry 2024/11/19-15:15:47.757088 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-00 2024/11/19-15:15:47.757140 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00036/att-00/compact.done) = No such file or directory: 0.131 sec on http probe failed: 4, retry 2024/11/19-15:15:47.790045 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-00 2024/11/19-15:15:47.790091 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00036/att-00/compact.done) = No such file or directory: 0.163 sec on http probe failed: 5, giveup 2024/11/19-15:15:47.790106 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00036/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.466 (submit 294.586 ms wait 0.163 s), input 1.215 MiB, send shutdown 2024/11/19-15:15:47.794275 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-00 2024/11/19-15:15:47.794475 139851179493120 [dcompact_etcd.cc:1773] job-00036/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:48.075832 139851179493120 [dcompact_etcd.cc:1163] job-00036/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 1.215 MiB 2024/11/19-15:15:48.085378 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-02 2024/11/19-15:15:48.085438 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00021/att-02/compact.done) = No such file or directory: 13.021 sec on http probe failed: 5, giveup 2024/11/19-15:15:48.085455 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00021/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 254.477 sec, real 13.300 (submit 274.353 ms wait 13.021 s), input 254.477 MiB, send shutdown 2024/11/19-15:15:48.093696 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00021/att-02 2024/11/19-15:15:48.093916 139851169007360 [dcompact_etcd.cc:1773] job-00021/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:48.095875 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00021) = success 2024/11/19-15:15:48.095883 139851169007360 [dcompact_etcd.cc:1773] job-00021/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:48.095901 139851169007360 (Original Log Time 2024/11/19-15:15:08.058147) EVENT_LOG_v1 {"time_micros": 1732000508058141, "job": 21, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4306, 4310, 4303, 4309], "score": 2.52844, "input_data_size": 259247796, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:48.095908 139851169007360 [compaction_job.cc:2437] [default] [JOB 21] Compacting 4@2 files to L3, score 2.53, subcompactions 1 : 1 2024/11/19-15:15:48.095916 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 19 Base level 2, inputs: [4306(62MB) 4310(61MB) 4303(62MB) 4309(61MB)] 2024/11/19-15:15:48.111871 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-01 2024/11/19-15:15:48.111948 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00036/att-01/compact.done) = No such file or directory: 0.030 sec on http probe failed: 1, retry 2024/11/19-15:15:48.139075 139851169007360 [compaction_job.cc:703] job-00021: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:15:48.139622 139851169007360 (Original Log Time 2024/11/19-15:15:48.095949) EVENT_LOG_v1 {"time_micros": 1732000548095933, "job": 21, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4306, 4310, 4303, 4309], "score": 2.52844, "input_data_size": 259247796, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:48.139632 139851169007360 (Original Log Time 2024/11/19-15:15:48.139136) [compaction_job.cc:2022] [default] [JOB 21] Compacted 4@2 files to L3 => 0 bytes 2024/11/19-15:15:48.139636 139851169007360 (Original Log Time 2024/11/19-15:15:48.139486) [compaction_job.cc:1224] [default] compacted to: files[2 70 75 75 0 0 0] max score 3.92 (129 files need compaction), MB/sec: 6009.5 rd, 0.0 wr, level 3, files in(4, 0) out(0 +0 blob) MB in(247.2, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 398876, records dropped: 398876 output_compression: Snappy 2024/11/19-15:15:48.139651 139851169007360 (Original Log Time 2024/11/19-15:15:48.139511) EVENT_LOG_v1 {"time_micros": 1732000548139499, "cf": "default", "job": 21, "event": "compaction_finished", "compaction_time_micros": 43140, "compaction_time_cpu_micros": 42941, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 398876, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 70, 75, 75, 0, 0, 0]} 2024/11/19-15:15:48.143511 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004193.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:48.143537 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000548143534, "job": 21, "event": "table_file_deletion", "file_number": 4193} 2024/11/19-15:15:48.144757 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-01 2024/11/19-15:15:48.144819 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00036/att-01/compact.done) = No such file or directory: 0.061 sec on http probe failed: 2, retry 2024/11/19-15:15:48.147322 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004194.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:48.147344 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000548147341, "job": 21, "event": "table_file_deletion", "file_number": 4194} 2024/11/19-15:15:48.151071 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004196.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:48.151091 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000548151089, "job": 21, "event": "table_file_deletion", "file_number": 4196} 2024/11/19-15:15:48.154899 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004197.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:48.154926 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000548154916, "job": 21, "event": "table_file_deletion", "file_number": 4197} 2024/11/19-15:15:48.158712 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004198.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:48.158732 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000548158730, "job": 21, "event": "table_file_deletion", "file_number": 4198} 2024/11/19-15:15:48.162542 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004200.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:48.162560 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000548162558, "job": 21, "event": "table_file_deletion", "file_number": 4200} 2024/11/19-15:15:48.166332 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004201.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:48.166351 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000548166349, "job": 21, "event": "table_file_deletion", "file_number": 4201} 2024/11/19-15:15:48.170404 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004202.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:48.170425 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000548170422, "job": 21, "event": "table_file_deletion", "file_number": 4202} 2024/11/19-15:15:48.170586 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004270.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:48.170596 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000548170595, "job": 21, "event": "table_file_deletion", "file_number": 4270} 2024/11/19-15:15:48.177195 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-01 2024/11/19-15:15:48.177264 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00036/att-01/compact.done) = No such file or directory: 0.093 sec on http probe failed: 3, retry 2024/11/19-15:15:48.178596 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004300.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:48.178616 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000548178614, "job": 21, "event": "table_file_deletion", "file_number": 4300} 2024/11/19-15:15:48.181609 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004301.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:15:48.181628 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000548181626, "job": 21, "event": "table_file_deletion", "file_number": 4301} 2024/11/19-15:15:48.181804 139851169007360 [compaction_job.cc:2437] [default] [JOB 37] Compacting 1@1 files to L2, score 3.92, subcompactions 1 : 1 2024/11/19-15:15:48.181812 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 35 Base level 1, inputs: [4364(13MB)] 2024/11/19-15:15:48.183155 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-00 2024/11/19-15:15:48.183227 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00031/att-00/compact.done) = No such file or directory: 2.717 sec on http probe failed: 4, retry 2024/11/19-15:15:48.209527 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-01 2024/11/19-15:15:48.209600 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00036/att-01/compact.done) = No such file or directory: 0.126 sec on http probe failed: 4, retry 2024/11/19-15:15:48.242834 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-01 2024/11/19-15:15:48.242911 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00036/att-01/compact.done) = No such file or directory: 0.158 sec on http probe failed: 5, giveup 2024/11/19-15:15:48.242937 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00036/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.448 (submit 281.258 ms wait 0.158 s), input 1.215 MiB, send shutdown 2024/11/19-15:15:48.251842 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-01 2024/11/19-15:15:48.252072 139851179493120 [dcompact_etcd.cc:1773] job-00036/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:48.508817 139851169007360 [dcompact_etcd.cc:1163] job-00037/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.326 sec, input = 12.700 MiB 2024/11/19-15:15:48.525701 139851179493120 [dcompact_etcd.cc:1163] job-00036/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.274 sec, input = 1.215 MiB 2024/11/19-15:15:48.564804 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-02 2024/11/19-15:15:48.564882 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00036/att-02/compact.done) = No such file or directory: 0.030 sec on http probe failed: 1, retry 2024/11/19-15:15:48.598970 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-02 2024/11/19-15:15:48.599024 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00036/att-02/compact.done) = No such file or directory: 0.064 sec on http probe failed: 2, retry 2024/11/19-15:15:48.631431 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-02 2024/11/19-15:15:48.631479 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00036/att-02/compact.done) = No such file or directory: 0.098 sec on http probe failed: 3, retry 2024/11/19-15:15:48.660224 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-02 2024/11/19-15:15:48.660271 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00036/att-02/compact.done) = No such file or directory: 0.130 sec on http probe failed: 4, retry 2024/11/19-15:15:48.689989 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-02 2024/11/19-15:15:48.690036 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00036/att-02/compact.done) = No such file or directory: 0.159 sec on http probe failed: 5, giveup 2024/11/19-15:15:48.690050 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00036/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.438 (submit 273.550 ms wait 0.159 s), input 1.215 MiB, send shutdown 2024/11/19-15:15:48.698413 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00036/att-02 2024/11/19-15:15:48.698632 139851179493120 [dcompact_etcd.cc:1773] job-00036/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:48.700570 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00036) = success 2024/11/19-15:15:48.700578 139851179493120 [dcompact_etcd.cc:1773] job-00036/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:48.700598 139851179493120 (Original Log Time 2024/11/19-15:15:47.323789) EVENT_LOG_v1 {"time_micros": 1732000547323785, "job": 36, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L2": [4425], "score": -1, "input_data_size": 92793, "oldest_snapshot_seqno": 109497446} 2024/11/19-15:15:48.700606 139851179493120 [compaction_job.cc:2437] [default] [JOB 36] Compacting 1@2 files to L3, score -1.00, subcompactions 1 : 1 2024/11/19-15:15:48.700613 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 33 Base level 2, inputs: [4425(90KB)] 2024/11/19-15:15:48.706386 139851179493120 [compaction_job.cc:703] job-00036: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:15:48.706842 139851179493120 (Original Log Time 2024/11/19-15:15:48.700645) EVENT_LOG_v1 {"time_micros": 1732000548700626, "job": 36, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L2": [4425], "score": -1, "input_data_size": 92793, "oldest_snapshot_seqno": 109497446} 2024/11/19-15:15:48.706850 139851179493120 (Original Log Time 2024/11/19-15:15:48.706440) [compaction_job.cc:2022] [default] [JOB 36] Compacted 1@2 files to L3 => 0 bytes 2024/11/19-15:15:48.706853 139851179493120 (Original Log Time 2024/11/19-15:15:48.706770) [compaction_job.cc:1224] [default] compacted to: files[2 70 74 75 0 0 0] max score 3.90 (128 files need compaction), MB/sec: 16.1 rd, 0.0 wr, level 3, files in(1, 0) out(0 +0 blob) MB in(0.1, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 53100, records dropped: 53100 output_compression: Snappy 2024/11/19-15:15:48.706859 139851179493120 (Original Log Time 2024/11/19-15:15:48.706792) EVENT_LOG_v1 {"time_micros": 1732000548706779, "cf": "default", "job": 36, "event": "compaction_finished", "compaction_time_micros": 5750, "compaction_time_cpu_micros": 5714, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 53100, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 70, 74, 75, 0, 0, 0]} 2024/11/19-15:15:48.706959 139834427442944 (Original Log Time 2024/11/19-15:15:46.740046) [db_impl_compaction_flush.cc:2043] [default] Manual compaction starting 2024/11/19-15:15:48.706979 139834427442944 (Original Log Time 2024/11/19-15:15:48.706976) [db_impl_compaction_flush.cc:2043] [default] Manual compaction starting 2024/11/19-15:15:48.706989 139851179493120 [compaction_job.cc:2437] [default] [JOB 38] Compacting 2@1 files to L2, score 3.90, subcompactions 1 : 1 2024/11/19-15:15:48.706999 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 36 Base level 1, inputs: [4368(4510KB) 4352(3503KB)] 2024/11/19-15:15:48.822301 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-00 2024/11/19-15:15:48.822414 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00031/att-00/compact.done) = No such file or directory: 3.361 sec on http probe failed: 5, giveup 2024/11/19-15:15:48.822438 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00031/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.702 sec, real 3.658 (submit 292.711 ms wait 3.361 s), input 31.702 MiB, send shutdown 2024/11/19-15:15:48.830509 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-00 2024/11/19-15:15:48.830759 139851127064320 [dcompact_etcd.cc:1773] job-00031/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:48.830964 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-00 2024/11/19-15:15:48.831025 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00037/att-00/compact.done) = No such file or directory: 0.318 sec on http probe failed: 1, retry 2024/11/19-15:15:49.013207 139851179493120 [dcompact_etcd.cc:1163] job-00038/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.306 sec, input = 7.519 MiB 2024/11/19-15:15:49.090471 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-00 2024/11/19-15:15:49.090536 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00037/att-00/compact.done) = No such file or directory: 0.576 sec on http probe failed: 2, retry 2024/11/19-15:15:49.117891 139851127064320 [dcompact_etcd.cc:1163] job-00031/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.287 sec, input = 31.702 MiB 2024/11/19-15:15:49.207088 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-00 2024/11/19-15:15:49.207189 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00038/att-00/compact.done) = No such file or directory: 0.188 sec on http probe failed: 1, retry 2024/11/19-15:15:49.246655 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-00 2024/11/19-15:15:49.246723 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00026/att-00/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:15:49.352136 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-00 2024/11/19-15:15:49.352221 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00037/att-00/compact.done) = No such file or directory: 0.836 sec on http probe failed: 3, retry 2024/11/19-15:15:49.362493 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-00 2024/11/19-15:15:49.362556 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00038/att-00/compact.done) = No such file or directory: 0.344 sec on http probe failed: 2, retry 2024/11/19-15:15:49.392181 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-00 2024/11/19-15:15:49.392239 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00027/att-00/compact.done) = No such file or directory: 7.008 sec on http probe failed: 2, retry 2024/11/19-15:15:49.421813 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-00 2024/11/19-15:15:49.421863 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00028/att-00/compact.done) = No such file or directory: 7.005 sec on http probe failed: 2, retry 2024/11/19-15:15:49.518068 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-00 2024/11/19-15:15:49.518125 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00038/att-00/compact.done) = No such file or directory: 0.500 sec on http probe failed: 3, retry 2024/11/19-15:15:49.614700 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-00 2024/11/19-15:15:49.614788 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00037/att-00/compact.done) = No such file or directory: 1.097 sec on http probe failed: 4, retry 2024/11/19-15:15:49.678660 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-00 2024/11/19-15:15:49.678752 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00038/att-00/compact.done) = No such file or directory: 0.655 sec on http probe failed: 4, retry 2024/11/19-15:15:49.835132 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-00 2024/11/19-15:15:49.835211 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00038/att-00/compact.done) = No such file or directory: 0.816 sec on http probe failed: 5, giveup 2024/11/19-15:15:49.835228 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00038/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 7.519 sec, real 1.128 (submit 305.755 ms wait 0.816 s), input 7.519 MiB, send shutdown 2024/11/19-15:15:49.842877 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-00 2024/11/19-15:15:49.843082 139851179493120 [dcompact_etcd.cc:1773] job-00038/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:49.877663 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-00 2024/11/19-15:15:49.877716 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00037/att-00/compact.done) = No such file or directory: 1.360 sec on http probe failed: 5, giveup 2024/11/19-15:15:49.877732 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00037/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 12.700 sec, real 1.695 (submit 326.483 ms wait 1.360 s), input 12.700 MiB, send shutdown 2024/11/19-15:15:49.885973 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-00 2024/11/19-15:15:49.886178 139851169007360 [dcompact_etcd.cc:1773] job-00037/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:49.896178 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-00 2024/11/19-15:15:49.896258 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00035/att-00/compact.done) = No such file or directory: 2.871 sec on http probe failed: 1, retry 2024/11/19-15:15:49.919460 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-01 2024/11/19-15:15:49.919537 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00031/att-01/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:15:50.100306 139851179493120 [dcompact_etcd.cc:1163] job-00038/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.257 sec, input = 7.519 MiB 2024/11/19-15:15:50.149700 139851169007360 [dcompact_etcd.cc:1163] job-00037/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.263 sec, input = 12.700 MiB 2024/11/19-15:15:50.293372 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-01 2024/11/19-15:15:50.293477 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00038/att-01/compact.done) = No such file or directory: 0.188 sec on http probe failed: 1, retry 2024/11/19-15:15:50.349038 139852349699840 [db_impl_write.cc:2181] [default] New memtable created with log file: #4427. Immutable memtables: 0. 2024/11/19-15:15:50.349612 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 39] Syncing log #4339 2024/11/19-15:15:50.360689 139850967680768 (Original Log Time 2024/11/19-15:15:50.349391) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:15:50.360701 139850967680768 [flush_job.cc:879] [default] [JOB 39] Flushing memtable with next log file: 4427 2024/11/19-15:15:50.360722 139850967680768 EVENT_LOG_v1 {"time_micros": 1732000550360716, "job": 39, "event": "flush_started", "num_memtables": 1, "num_entries": 1509358, "num_deletes": 0, "total_data_size": 465222577, "memory_usage": 539753864, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:15:50.360726 139850967680768 [flush_job.cc:910] [default] [JOB 39] Level-0 flush table #4428: started 2024/11/19-15:15:50.361089 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000002.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004428.sst in ConvertToSST, mem_size = 514.749 M, time = 354.671 us 2024/11/19-15:15:50.452604 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-01 2024/11/19-15:15:50.452697 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00038/att-01/compact.done) = No such file or directory: 0.344 sec on http probe failed: 2, retry 2024/11/19-15:15:50.475495 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-01 2024/11/19-15:15:50.475569 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00037/att-01/compact.done) = No such file or directory: 0.318 sec on http probe failed: 1, retry 2024/11/19-15:15:50.563447 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-01 2024/11/19-15:15:50.563515 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00031/att-01/compact.done) = No such file or directory: 1.436 sec on http probe failed: 2, retry 2024/11/19-15:15:50.611584 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-01 2024/11/19-15:15:50.611643 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00038/att-01/compact.done) = No such file or directory: 0.503 sec on http probe failed: 3, retry 2024/11/19-15:15:50.737517 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-01 2024/11/19-15:15:50.737694 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00037/att-01/compact.done) = No such file or directory: 0.580 sec on http probe failed: 2, retry 2024/11/19-15:15:50.769862 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-01 2024/11/19-15:15:50.769949 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00038/att-01/compact.done) = No such file or directory: 0.662 sec on http probe failed: 4, retry 2024/11/19-15:15:50.780673 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004428.sst): fsize = 514.751 M, time(ms): open: 0.043, seek: 0.027, finish: 0.039, meta: 0.012, Flush: 0.013, sync: 419.426, close: 0.008, all: 419.568 2024/11/19-15:15:50.781945 139850967680768 [flush_job.cc:1083] [default] [JOB 39] Flush lasted 421256 microseconds, and 113447 cpu microseconds. 2024/11/19-15:15:50.803605 139850967680768 (Original Log Time 2024/11/19-15:15:50.780715) [flush_job.cc:1029] [default] [JOB 39] Level-0 flush table #4428: 539755290 bytes OK (needs compaction) 2024/11/19-15:15:50.803617 139850967680768 (Original Log Time 2024/11/19-15:15:50.781963) [memtable_list.cc:560] [default] Level-0 commit table #4428 started 2024/11/19-15:15:50.803621 139850967680768 (Original Log Time 2024/11/19-15:15:50.803534) [memtable_list.cc:759] [default] Level-0 commit table #4428: memtable #1 done 2024/11/19-15:15:50.803624 139850967680768 (Original Log Time 2024/11/19-15:15:50.803561) EVENT_LOG_v1 {"time_micros": 1732000550803553, "job": 39, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [3, 70, 74, 75, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:15:50.803627 139850967680768 (Original Log Time 2024/11/19-15:15:50.803577) [db_impl_compaction_flush.cc:310] [default] Level summary: files[3 70 74 75 0 0 0] max score 3.88 (127 files need compaction) 2024/11/19-15:15:50.929615 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-01 2024/11/19-15:15:50.929710 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00038/att-01/compact.done) = No such file or directory: 0.820 sec on http probe failed: 5, giveup 2024/11/19-15:15:50.929726 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00038/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 7.519 sec, real 1.087 (submit 257.128 ms wait 0.820 s), input 7.519 MiB, send shutdown 2024/11/19-15:15:50.934768 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-01 2024/11/19-15:15:50.946471 139851179493120 [dcompact_etcd.cc:1773] job-00038/att-01: CleanFiles 0.012 sec : 0.000 2024/11/19-15:15:51.000211 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-01 2024/11/19-15:15:51.000266 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00037/att-01/compact.done) = No such file or directory: 0.842 sec on http probe failed: 3, retry 2024/11/19-15:15:51.206157 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-01 2024/11/19-15:15:51.206260 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00031/att-01/compact.done) = No such file or directory: 2.080 sec on http probe failed: 3, retry 2024/11/19-15:15:51.249753 139851179493120 [dcompact_etcd.cc:1163] job-00038/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.303 sec, input = 7.519 MiB 2024/11/19-15:15:51.251836 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-00 2024/11/19-15:15:51.251893 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00026/att-00/compact.done) = No such file or directory: 9.015 sec on http probe failed: 3, retry 2024/11/19-15:15:51.262353 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-01 2024/11/19-15:15:51.262408 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00037/att-01/compact.done) = No such file or directory: 1.105 sec on http probe failed: 4, retry 2024/11/19-15:15:51.401058 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-00 2024/11/19-15:15:51.401117 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00027/att-00/compact.done) = No such file or directory: 9.017 sec on http probe failed: 3, retry 2024/11/19-15:15:51.429326 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-00 2024/11/19-15:15:51.429378 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00028/att-00/compact.done) = No such file or directory: 9.010 sec on http probe failed: 3, retry 2024/11/19-15:15:51.446633 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-02 2024/11/19-15:15:51.446696 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00038/att-02/compact.done) = No such file or directory: 0.188 sec on http probe failed: 1, retry 2024/11/19-15:15:51.523845 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-01 2024/11/19-15:15:51.523973 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00037/att-01/compact.done) = No such file or directory: 1.367 sec on http probe failed: 5, giveup 2024/11/19-15:15:51.524001 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00037/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 12.700 sec, real 1.638 (submit 263.451 ms wait 1.367 s), input 12.700 MiB, send shutdown 2024/11/19-15:15:51.531665 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-01 2024/11/19-15:15:51.531936 139851169007360 [dcompact_etcd.cc:1773] job-00037/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:51.606284 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-02 2024/11/19-15:15:51.606406 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00038/att-02/compact.done) = No such file or directory: 0.347 sec on http probe failed: 2, retry 2024/11/19-15:15:51.762964 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-02 2024/11/19-15:15:51.763050 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00038/att-02/compact.done) = No such file or directory: 0.507 sec on http probe failed: 3, retry 2024/11/19-15:15:51.826688 139851169007360 [dcompact_etcd.cc:1163] job-00037/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.295 sec, input = 12.700 MiB 2024/11/19-15:15:51.850238 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-01 2024/11/19-15:15:51.850305 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00031/att-01/compact.done) = No such file or directory: 2.722 sec on http probe failed: 4, retry 2024/11/19-15:15:51.904370 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-00 2024/11/19-15:15:51.904441 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00035/att-00/compact.done) = No such file or directory: 4.876 sec on http probe failed: 2, retry 2024/11/19-15:15:51.921400 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-02 2024/11/19-15:15:51.921525 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00038/att-02/compact.done) = No such file or directory: 0.664 sec on http probe failed: 4, retry 2024/11/19-15:15:52.081399 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-02 2024/11/19-15:15:52.081507 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00038/att-02/compact.done) = No such file or directory: 0.822 sec on http probe failed: 5, giveup 2024/11/19-15:15:52.081524 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00038/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 7.519 sec, real 1.135 (submit 303.198 ms wait 0.822 s), input 7.519 MiB, send shutdown 2024/11/19-15:15:52.087057 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00038/att-02 2024/11/19-15:15:52.087278 139851179493120 [dcompact_etcd.cc:1773] job-00038/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:52.089075 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00038) = success 2024/11/19-15:15:52.089090 139851179493120 [dcompact_etcd.cc:1773] job-00038/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:52.089118 139851179493120 (Original Log Time 2024/11/19-15:15:48.707014) EVENT_LOG_v1 {"time_micros": 1732000548707008, "job": 38, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4368, 4352], "score": 3.89624, "input_data_size": 8206269, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:52.089125 139851179493120 [compaction_job.cc:2437] [default] [JOB 38] Compacting 2@1 files to L2, score 3.90, subcompactions 1 : 1 2024/11/19-15:15:52.089133 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 36 Base level 1, inputs: [4368(4510KB) 4352(3503KB)] 2024/11/19-15:15:52.089369 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004429.sst) /storage/mytopling/..//tmp/Topling-82038-AvXLfG 2024/11/19-15:15:52.134153 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004429.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0016 GB, waited 0.000 sec, Key+Value = 0.008 GB 2024/11/19-15:15:52.143544 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004429.sst): index pass time = 0.01's, 230.687'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 180000 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0022 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 2024/11/19-15:15:52.145367 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004429.sst): first pass time = 0.06's, 140.818'MB/sec 2024/11/19-15:15:52.145405 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004429.sst): old prealloc_size = 9026895, real_size = 4076656 2024/11/19-15:15:52.145492 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004429.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 180000 prefix = 4 raw-key = 0.0022 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 raw-val = 0.0036 GB zip-val = 0.0037 GB avg-val = 19.80 avg-zval = 20.71 2024/11/19-15:15:52.145529 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004429.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.011 sec, Key+Value = 0.008 GB 2024/11/19-15:15:52.153270 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-02 2024/11/19-15:15:52.153336 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00037/att-02/compact.done) = No such file or directory: 0.318 sec on http probe failed: 1, retry 2024/11/19-15:15:52.164474 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004429.sst): second pass time = 0.00's,12774193.548'MB/sec, value only(45.2% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.02's, 211.721'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3686.815'MB/sec (index lex order gen) rebuild zvType time = 0.01's, 7.476'MB/sec write SST data time = 0.01's, 311.435'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.004'GB zip my value throughput = 12774193.548'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 180000 avg-key = 24.00 avg-zkey = 1.69 avg-val = 19.80 avg-zval = 20.71 usrkeys = 180000 avg-key = 16.00 avg-zkey = 1.69 avg-val = 19.80 avg-zval = 20.71 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 443 UnZipSize{ index = 0.0029 GB value = 0.0036 GB dict = 0.00 MB all = 0.0079 GB } __ZipSize{ index = 0.0003 GB value = 0.0037 GB dict = 0.00 MB all = 0.0040 GB } UnZip/Zip{ index = 9.4737 value = 0.9561 dict = 0.00 all = 1.9543 } Zip/UnZip{ index = 0.1056 value = 1.0459 dict = 0.00 all = 0.5117 } ---------------------------- total value len = 1.476183 GB avg = 0.500 KB (by entry num) total key len = 0.071420 GB avg = 0.024 KB total ukey len = 0.047814 GB avg = 0.016 KB total ukey num = 0.002950680 Billion total entry num = 0.002950680 Billion write speed all = 26.945848079 MB/sec (with seq num) write speed all = 26.534845736 MB/sec (without seq num) 2024/11/19-15:15:52.165206 139851179493120 [compaction_job.cc:1948] [default] [JOB 38] Generated table #4429: 180000 keys, 4034161 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:52.165254 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000552165230, "cf_name": "default", "job": 38, "event": "table_file_creation", "file_number": 4429, "file_size": 4034161, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 3727456, "index_size": 304000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4320000, "raw_average_key_size": 24, "raw_value_size": 3564000, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 180000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000552, "oldest_key_time": 0, "file_creation_time": 1732000552, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4429, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:52.166579 139851179493120 [compaction_job.cc:703] job-00038: subcompact[0], size: 0.004034 G, files: /storage/mytopling/.rocksdb [004429] 2024/11/19-15:15:52.167440 139851179493120 (Original Log Time 2024/11/19-15:15:52.089166) EVENT_LOG_v1 {"time_micros": 1732000552089147, "job": 38, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4368, 4352], "score": 3.89624, "input_data_size": 8206269, "oldest_snapshot_seqno": -1} 2024/11/19-15:15:52.167450 139851179493120 (Original Log Time 2024/11/19-15:15:52.166989) [compaction_job.cc:2022] [default] [JOB 38] Compacted 2@1 files to L2 => 4034161 bytes 2024/11/19-15:15:52.167453 139851179493120 (Original Log Time 2024/11/19-15:15:52.167332) [compaction_job.cc:1224] [default] compacted to: files[3 68 75 75 0 0 0] max score 3.88 (128 files need compaction), MB/sec: 106.0 rd, 52.1 wr, level 2, files in(2, 0) out(1 +0 blob) MB in(7.8, 0.0 +0.0 blob) out(3.8 +0.0 blob), read-write-amplify(1.5) write-amplify(0.5) OK, records in: 180000, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:52.167469 139851179493120 (Original Log Time 2024/11/19-15:15:52.167354) EVENT_LOG_v1 {"time_micros": 1732000552167343, "cf": "default", "job": 38, "event": "compaction_finished", "compaction_time_micros": 77422, "compaction_time_cpu_micros": 77018, "output_level": 2, "num_output_files": 1, "total_output_size": 4034161, "num_input_records": 180000, "num_output_records": 180000, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 68, 75, 75, 0, 0, 0]} 2024/11/19-15:15:52.167540 139851179493120 [compaction_job.cc:2437] [default] [JOB 40] Compacting 4@1 files to L2, score 3.88, subcompactions 1 : 1 2024/11/19-15:15:52.167548 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 38 Base level 1, inputs: [4356(31MB) 4363(31MB) 4370(31MB) 4373(31MB)] 2024/11/19-15:15:52.416268 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-02 2024/11/19-15:15:52.416356 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00037/att-02/compact.done) = No such file or directory: 0.581 sec on http probe failed: 2, retry 2024/11/19-15:15:52.469542 139851179493120 [dcompact_etcd.cc:1163] job-00040/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.302 sec, input = 127.821 MiB 2024/11/19-15:15:52.493181 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-01 2024/11/19-15:15:52.493260 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00031/att-01/compact.done) = No such file or directory: 3.367 sec on http probe failed: 5, giveup 2024/11/19-15:15:52.493277 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00031/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.702 sec, real 3.662 (submit 287.031 ms wait 3.367 s), input 31.702 MiB, send shutdown 2024/11/19-15:15:52.501278 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-01 2024/11/19-15:15:52.501495 139851127064320 [dcompact_etcd.cc:1773] job-00031/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:52.678990 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-02 2024/11/19-15:15:52.679077 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00037/att-02/compact.done) = No such file or directory: 0.844 sec on http probe failed: 3, retry 2024/11/19-15:15:52.787510 139851127064320 [dcompact_etcd.cc:1163] job-00031/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.286 sec, input = 31.702 MiB 2024/11/19-15:15:52.942707 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-02 2024/11/19-15:15:52.942795 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00037/att-02/compact.done) = No such file or directory: 1.106 sec on http probe failed: 4, retry 2024/11/19-15:15:53.206357 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-02 2024/11/19-15:15:53.206466 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00037/att-02/compact.done) = No such file or directory: 1.370 sec on http probe failed: 5, giveup 2024/11/19-15:15:53.206491 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00037/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 12.700 sec, real 1.674 (submit 294.648 ms wait 1.370 s), input 12.700 MiB, send shutdown 2024/11/19-15:15:53.211579 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00037/att-02 2024/11/19-15:15:53.211851 139851169007360 [dcompact_etcd.cc:1773] job-00037/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:53.213672 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00037) = success 2024/11/19-15:15:53.213707 139851169007360 [dcompact_etcd.cc:1773] job-00037/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:15:53.213732 139851169007360 (Original Log Time 2024/11/19-15:15:48.181825) EVENT_LOG_v1 {"time_micros": 1732000548181818, "job": 37, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4364], "score": 3.92271, "input_data_size": 14209155, "oldest_snapshot_seqno": 109580472} 2024/11/19-15:15:53.213743 139851169007360 [compaction_job.cc:2437] [default] [JOB 37] Compacting 1@1 files to L2, score 3.92, subcompactions 1 : 1 2024/11/19-15:15:53.213754 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 35 Base level 1, inputs: [4364(13MB)] 2024/11/19-15:15:53.214054 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004430.sst) /storage/mytopling/..//tmp/Topling-82038-ZIQVt5 2024/11/19-15:15:53.256583 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-00 2024/11/19-15:15:53.256680 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00026/att-00/compact.done) = No such file or directory: 11.020 sec on http probe failed: 4, retry 2024/11/19-15:15:53.281839 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004430.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.013 GB 2024/11/19-15:15:53.281980 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004430.sst): index pass time = 0.00's,13627.975'MB/sec index type = UintIndex_AllOne usrkeys = 180000 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0014 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:15:53.297187 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004430.sst): first pass time = 0.08's, 160.221'MB/sec 2024/11/19-15:15:53.297237 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004430.sst): old prealloc_size = 15630070, real_size = 10302232 2024/11/19-15:15:53.297259 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004430.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 180000 prefix = 4 raw-key = 0.0014 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0097 GB zip-val = 0.0103 GB avg-val = 53.98 avg-zval = 56.98 2024/11/19-15:15:53.312617 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004430.sst): second pass time = 0.00's,44166572.727'MB/sec, value only(73.0% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.02's, 667.631'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.281'MB/sec (index lex order gen) rebuild zvType time = 0.01's, 7.426'MB/sec write SST data time = 0.01's, 1104.536'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.010'GB zip my value throughput = 44166572.727'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 180000 avg-key = 20.00 avg-zkey = 0.00 avg-val = 53.98 avg-zval = 56.98 usrkeys = 180000 avg-key = 12.00 avg-zkey = 0.00 avg-val = 53.98 avg-zval = 56.98 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0022 GB value = 0.0097 GB dict = 0.00 MB all = 0.0133 GB } __ZipSize{ index = 0.0000 GB value = 0.0103 GB dict = 0.00 MB all = 0.0103 GB } UnZip/Zip{ index =11250.0000 value = 0.9473 dict = 0.00 all = 1.2982 } Zip/UnZip{ index = 0.0001 value = 1.0556 dict = 0.00 all = 0.7703 } ---------------------------- total value len = 1.485900 GB avg = 0.475 KB (by entry num) total key len = 0.075020 GB avg = 0.024 KB total ukey len = 0.049974 GB avg = 0.016 KB total ukey num = 0.003130680 Billion total entry num = 0.003130680 Billion write speed all = 26.645057287 MB/sec (with seq num) write speed all = 26.217529170 MB/sec (without seq num) 2024/11/19-15:15:53.313691 139851169007360 [compaction_job.cc:1948] [default] [JOB 37] Generated table #4430: 180000 keys, 10258164 bytes (need compaction), temperature: kUnknown 2024/11/19-15:15:53.313742 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000553313719, "cf_name": "default", "job": 37, "event": "table_file_creation", "file_number": 4430, "file_size": 10258164, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 10256864, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3600000, "raw_average_key_size": 20, "raw_value_size": 9716646, "raw_average_value_size": 53, "num_data_blocks": 1, "num_entries": 180000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000553, "oldest_key_time": 0, "file_creation_time": 1732000553, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4430, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:15:53.316619 139851169007360 [compaction_job.cc:703] job-00037: subcompact[0], size: 0.010258 G, files: /storage/mytopling/.rocksdb [004430] 2024/11/19-15:15:53.317340 139851169007360 (Original Log Time 2024/11/19-15:15:53.213793) EVENT_LOG_v1 {"time_micros": 1732000553213770, "job": 37, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4364], "score": 3.92271, "input_data_size": 14209155, "oldest_snapshot_seqno": 109580472} 2024/11/19-15:15:53.317350 139851169007360 (Original Log Time 2024/11/19-15:15:53.316880) [compaction_job.cc:2022] [default] [JOB 37] Compacted 1@1 files to L2 => 10258164 bytes 2024/11/19-15:15:53.317353 139851169007360 (Original Log Time 2024/11/19-15:15:53.317248) [compaction_job.cc:1224] [default] compacted to: files[3 67 76 75 0 0 0] max score 3.63 (125 files need compaction), MB/sec: 138.2 rd, 99.8 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(13.6, 0.0 +0.0 blob) out(9.8 +0.0 blob), read-write-amplify(1.7) write-amplify(0.7) OK, records in: 180000, records dropped: 0 output_compression: Snappy 2024/11/19-15:15:53.317369 139851169007360 (Original Log Time 2024/11/19-15:15:53.317271) EVENT_LOG_v1 {"time_micros": 1732000553317259, "cf": "default", "job": 37, "event": "compaction_finished", "compaction_time_micros": 102836, "compaction_time_cpu_micros": 102260, "output_level": 2, "num_output_files": 1, "total_output_size": 10258164, "num_input_records": 180000, "num_output_records": 180000, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 67, 76, 75, 0, 0, 0]} 2024/11/19-15:15:53.317444 139851169007360 [compaction_job.cc:2437] [default] [JOB 41] Compacting 4@1 files to L2, score 3.63, subcompactions 1 : 1 2024/11/19-15:15:53.317452 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 39 Base level 1, inputs: [4378(31MB) 4382(31MB) 4387(31MB) 4391(20MB)] 2024/11/19-15:15:53.406035 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-00 2024/11/19-15:15:53.406111 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00027/att-00/compact.done) = No such file or directory: 11.026 sec on http probe failed: 4, retry 2024/11/19-15:15:53.437326 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-00 2024/11/19-15:15:53.437391 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00028/att-00/compact.done) = No such file or directory: 11.018 sec on http probe failed: 4, retry 2024/11/19-15:15:53.589467 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-02 2024/11/19-15:15:53.589575 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00031/att-02/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:15:53.629307 139851169007360 [dcompact_etcd.cc:1163] job-00041/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.311 sec, input = 116.546 MiB 2024/11/19-15:15:53.912905 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-00 2024/11/19-15:15:53.913040 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00035/att-00/compact.done) = No such file or directory: 6.884 sec on http probe failed: 3, retry 2024/11/19-15:15:54.228322 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-02 2024/11/19-15:15:54.228399 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00031/att-02/compact.done) = No such file or directory: 1.436 sec on http probe failed: 2, retry 2024/11/19-15:15:54.870890 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-02 2024/11/19-15:15:54.870975 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00031/att-02/compact.done) = No such file or directory: 2.075 sec on http probe failed: 3, retry 2024/11/19-15:15:55.262824 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-00 2024/11/19-15:15:55.262917 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00026/att-00/compact.done) = No such file or directory: 13.025 sec on http probe failed: 5, giveup 2024/11/19-15:15:55.262956 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00026/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 223.779 sec, real 13.341 (submit 309.911 ms wait 13.025 s), input 223.779 MiB, send shutdown 2024/11/19-15:15:55.270861 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-00 2024/11/19-15:15:55.271133 139851148035840 [dcompact_etcd.cc:1773] job-00026/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:55.414484 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-00 2024/11/19-15:15:55.414563 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00027/att-00/compact.done) = No such file or directory: 13.031 sec on http probe failed: 5, giveup 2024/11/19-15:15:55.414582 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00027/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.637 sec, real 13.358 (submit 318.662 ms wait 13.031 s), input 255.637 MiB, send shutdown 2024/11/19-15:15:55.422968 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-00 2024/11/19-15:15:55.423201 139851158521600 [dcompact_etcd.cc:1773] job-00027/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:55.444875 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-00 2024/11/19-15:15:55.444942 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00028/att-00/compact.done) = No such file or directory: 13.026 sec on http probe failed: 5, giveup 2024/11/19-15:15:55.444959 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00028/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.641 sec, real 13.351 (submit 317.727 ms wait 13.026 s), input 255.641 MiB, send shutdown 2024/11/19-15:15:55.452778 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-00 2024/11/19-15:15:55.452983 139851137550080 [dcompact_etcd.cc:1773] job-00028/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:55.510174 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-02 2024/11/19-15:15:55.510251 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00031/att-02/compact.done) = No such file or directory: 2.718 sec on http probe failed: 4, retry 2024/11/19-15:15:55.598516 139851148035840 [dcompact_etcd.cc:1163] job-00026/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.327 sec, input = 223.779 MiB 2024/11/19-15:15:55.673647 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-00 2024/11/19-15:15:55.673732 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00040/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:15:55.714446 139851158521600 [dcompact_etcd.cc:1163] job-00027/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.291 sec, input = 255.637 MiB 2024/11/19-15:15:55.762748 139851137550080 [dcompact_etcd.cc:1163] job-00028/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.310 sec, input = 255.641 MiB 2024/11/19-15:15:55.921176 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-00 2024/11/19-15:15:55.921343 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00035/att-00/compact.done) = No such file or directory: 8.893 sec on http probe failed: 4, retry 2024/11/19-15:15:56.153531 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-02 2024/11/19-15:15:56.153610 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00031/att-02/compact.done) = No such file or directory: 3.357 sec on http probe failed: 5, giveup 2024/11/19-15:15:56.153628 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00031/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.702 sec, real 3.652 (submit 285.934 ms wait 3.357 s), input 31.702 MiB, send shutdown 2024/11/19-15:15:56.158097 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00031/att-02 2024/11/19-15:15:56.158320 139851127064320 [dcompact_etcd.cc:1773] job-00031/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:56.161051 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00031) = success 2024/11/19-15:15:56.161059 139851127064320 [dcompact_etcd.cc:1773] job-00031/att-03: CleanFiles 0.003 sec : 0.000 2024/11/19-15:15:56.161078 139851127064320 (Original Log Time 2024/11/19-15:15:45.163675) EVENT_LOG_v1 {"time_micros": 1732000545163670, "job": 31, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [3809], "score": 2.19662, "input_data_size": 32758148, "oldest_snapshot_seqno": 109291688} 2024/11/19-15:15:56.161085 139851127064320 [compaction_job.cc:2437] [default] [JOB 31] Compacting 1@2 files to L3, score 2.20, subcompactions 1 : 1 2024/11/19-15:15:56.161092 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 29 Base level 2, inputs: [3809(31MB)] 2024/11/19-15:15:56.164702 139851127064320 [compaction_job.cc:703] job-00031: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:15:56.165193 139851127064320 (Original Log Time 2024/11/19-15:15:56.161121) EVENT_LOG_v1 {"time_micros": 1732000556161104, "job": 31, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [3809], "score": 2.19662, "input_data_size": 32758148, "oldest_snapshot_seqno": 109291688} 2024/11/19-15:15:56.165203 139851127064320 (Original Log Time 2024/11/19-15:15:56.164758) [compaction_job.cc:2022] [default] [JOB 31] Compacted 1@2 files to L3 => 0 bytes 2024/11/19-15:15:56.165206 139851127064320 (Original Log Time 2024/11/19-15:15:56.165104) [compaction_job.cc:1224] [default] compacted to: files[3 67 75 75 0 0 0] max score 3.40 (121 files need compaction), MB/sec: 9127.4 rd, 0.0 wr, level 3, files in(1, 0) out(0 +0 blob) MB in(31.2, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 32058, records dropped: 32058 output_compression: Snappy 2024/11/19-15:15:56.165221 139851127064320 (Original Log Time 2024/11/19-15:15:56.165124) EVENT_LOG_v1 {"time_micros": 1732000556165113, "cf": "default", "job": 31, "event": "compaction_finished", "compaction_time_micros": 3589, "compaction_time_cpu_micros": 3553, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 32058, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 67, 75, 75, 0, 0, 0]} 2024/11/19-15:15:56.165297 139851127064320 [compaction_job.cc:2437] [default] [JOB 42] Compacting 4@1 files to L2, score 3.40, subcompactions 1 : 1 2024/11/19-15:15:56.165305 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 40 Base level 1, inputs: [4394(31MB) 4396(31MB) 4398(31MB) 4400(31MB)] 2024/11/19-15:15:56.446403 139851127064320 [dcompact_etcd.cc:1163] job-00042/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 126.879 MiB 2024/11/19-15:15:56.548370 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-00 2024/11/19-15:15:56.548458 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00041/att-00/compact.done) = No such file or directory: 2.914 sec on http probe failed: 1, retry 2024/11/19-15:15:57.683017 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-00 2024/11/19-15:15:57.683110 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00040/att-00/compact.done) = No such file or directory: 5.204 sec on http probe failed: 2, retry 2024/11/19-15:15:57.930749 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-00 2024/11/19-15:15:57.930850 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00035/att-00/compact.done) = No such file or directory: 10.901 sec on http probe failed: 5, giveup 2024/11/19-15:15:57.930870 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00035/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 114.819 sec, real 11.190 (submit 279.747 ms wait 10.901 s), input 114.819 MiB, send shutdown 2024/11/19-15:15:57.939709 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-00 2024/11/19-15:15:57.940004 139850978166528 [dcompact_etcd.cc:1773] job-00035/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:15:58.216630 139850978166528 [dcompact_etcd.cc:1163] job-00035/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.277 sec, input = 114.819 MiB 2024/11/19-15:15:58.557429 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-00 2024/11/19-15:15:58.557507 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00041/att-00/compact.done) = No such file or directory: 4.919 sec on http probe failed: 2, retry 2024/11/19-15:15:59.626549 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-00 2024/11/19-15:15:59.626646 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00042/att-00/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:15:59.690839 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-00 2024/11/19-15:15:59.690904 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00040/att-00/compact.done) = No such file or directory: 7.214 sec on http probe failed: 3, retry 2024/11/19-15:16:00.567285 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-00 2024/11/19-15:16:00.567377 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00041/att-00/compact.done) = No such file or directory: 6.928 sec on http probe failed: 3, retry 2024/11/19-15:16:00.608082 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-01 2024/11/19-15:16:00.608170 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00026/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:16:00.721131 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-01 2024/11/19-15:16:00.721221 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00027/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:16:00.773185 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-01 2024/11/19-15:16:00.773289 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00028/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:16:01.095833 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-01 2024/11/19-15:16:01.095912 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00035/att-01/compact.done) = No such file or directory: 2.871 sec on http probe failed: 1, retry 2024/11/19-15:16:01.636459 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-00 2024/11/19-15:16:01.636541 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00042/att-00/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:16:01.696283 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-00 2024/11/19-15:16:01.696354 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00040/att-00/compact.done) = No such file or directory: 9.221 sec on http probe failed: 4, retry 2024/11/19-15:16:02.572819 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-00 2024/11/19-15:16:02.572933 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00041/att-00/compact.done) = No such file or directory: 8.938 sec on http probe failed: 4, retry 2024/11/19-15:16:02.614280 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-01 2024/11/19-15:16:02.614351 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00026/att-01/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:16:02.730533 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-01 2024/11/19-15:16:02.730617 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00027/att-01/compact.done) = No such file or directory: 7.007 sec on http probe failed: 2, retry 2024/11/19-15:16:02.781138 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-01 2024/11/19-15:16:02.781200 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00028/att-01/compact.done) = No such file or directory: 7.011 sec on http probe failed: 2, retry 2024/11/19-15:16:03.104554 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-01 2024/11/19-15:16:03.104666 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00035/att-01/compact.done) = No such file or directory: 4.879 sec on http probe failed: 2, retry 2024/11/19-15:16:03.644495 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-00 2024/11/19-15:16:03.644592 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00042/att-00/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:16:03.704029 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-00 2024/11/19-15:16:03.704157 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00040/att-00/compact.done) = No such file or directory: 11.227 sec on http probe failed: 5, giveup 2024/11/19-15:16:03.704186 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00040/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.821 sec, real 11.536 (submit 301.533 ms wait 11.227 s), input 127.821 MiB, send shutdown 2024/11/19-15:16:03.713441 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-00 2024/11/19-15:16:03.713687 139851179493120 [dcompact_etcd.cc:1773] job-00040/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:04.025077 139851179493120 [dcompact_etcd.cc:1163] job-00040/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.311 sec, input = 127.821 MiB 2024/11/19-15:16:04.581909 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-00 2024/11/19-15:16:04.582019 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00041/att-00/compact.done) = No such file or directory: 10.944 sec on http probe failed: 5, giveup 2024/11/19-15:16:04.582038 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00041/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 116.546 sec, real 11.264 (submit 311.388 ms wait 10.944 s), input 116.546 MiB, send shutdown 2024/11/19-15:16:04.590763 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-00 2024/11/19-15:16:04.591006 139851169007360 [dcompact_etcd.cc:1773] job-00041/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:04.622882 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-01 2024/11/19-15:16:04.622951 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00026/att-01/compact.done) = No such file or directory: 9.016 sec on http probe failed: 3, retry 2024/11/19-15:16:04.738857 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-01 2024/11/19-15:16:04.738953 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00027/att-01/compact.done) = No such file or directory: 9.016 sec on http probe failed: 3, retry 2024/11/19-15:16:04.789828 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-01 2024/11/19-15:16:04.789899 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00028/att-01/compact.done) = No such file or directory: 9.019 sec on http probe failed: 3, retry 2024/11/19-15:16:04.839978 139851169007360 [dcompact_etcd.cc:1163] job-00041/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.249 sec, input = 116.546 MiB 2024/11/19-15:16:05.111158 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-01 2024/11/19-15:16:05.111239 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00035/att-01/compact.done) = No such file or directory: 6.888 sec on http probe failed: 3, retry 2024/11/19-15:16:05.653372 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-00 2024/11/19-15:16:05.653476 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00042/att-00/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:16:05.855215 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4431. Immutable memtables: 0. 2024/11/19-15:16:05.855781 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 43] Syncing log #4427 2024/11/19-15:16:05.855872 139850957195008 (Original Log Time 2024/11/19-15:16:05.855584) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:16:05.855880 139850957195008 [flush_job.cc:879] [default] [JOB 43] Flushing memtable with next log file: 4431 2024/11/19-15:16:05.855902 139850957195008 EVENT_LOG_v1 {"time_micros": 1732000565855895, "job": 43, "event": "flush_started", "num_memtables": 1, "num_entries": 1181119, "num_deletes": 0, "total_data_size": 498632619, "memory_usage": 536870280, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:16:05.855905 139850957195008 [flush_job.cc:910] [default] [JOB 43] Level-0 flush table #4432: started 2024/11/19-15:16:05.856200 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000003.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004432.sst in ConvertToSST, mem_size = 511.999 M, time = 286.411 us 2024/11/19-15:16:06.499580 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004432.sst): fsize = 512.001 M, time(ms): open: 0.039, seek: 0.025, finish: 0.038, meta: 0.012, Flush: 0.010, sync: 643.224, close: 0.014, all: 643.362 2024/11/19-15:16:06.500679 139850957195008 [flush_job.cc:1083] [default] [JOB 43] Flush lasted 644806 microseconds, and 397768 cpu microseconds. 2024/11/19-15:16:06.520578 139850957195008 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:16:06.520633 139850957195008 (Original Log Time 2024/11/19-15:16:06.499628) [flush_job.cc:1029] [default] [JOB 43] Level-0 flush table #4432: 536871706 bytes OK (needs compaction) 2024/11/19-15:16:06.520639 139850957195008 (Original Log Time 2024/11/19-15:16:06.500697) [memtable_list.cc:560] [default] Level-0 commit table #4432 started 2024/11/19-15:16:06.520642 139850957195008 (Original Log Time 2024/11/19-15:16:06.520545) [memtable_list.cc:759] [default] Level-0 commit table #4432: memtable #1 done 2024/11/19-15:16:06.520645 139850957195008 (Original Log Time 2024/11/19-15:16:06.520572) EVENT_LOG_v1 {"time_micros": 1732000566520564, "job": 43, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [4, 67, 75, 75, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:16:06.520648 139850957195008 (Original Log Time 2024/11/19-15:16:06.520604) [db_impl_compaction_flush.cc:310] [default] Level summary: files[4 67 75 75 0 0 0] max score 3.15 (118 files need compaction) 2024/11/19-15:16:06.632177 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-01 2024/11/19-15:16:06.632269 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00026/att-01/compact.done) = No such file or directory: 11.025 sec on http probe failed: 4, retry 2024/11/19-15:16:06.744625 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-01 2024/11/19-15:16:06.744707 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00027/att-01/compact.done) = No such file or directory: 11.025 sec on http probe failed: 4, retry 2024/11/19-15:16:06.798338 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-01 2024/11/19-15:16:06.798458 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00028/att-01/compact.done) = No such file or directory: 11.027 sec on http probe failed: 4, retry 2024/11/19-15:16:07.121717 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-01 2024/11/19-15:16:07.121805 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00035/att-01/compact.done) = No such file or directory: 8.895 sec on http probe failed: 4, retry 2024/11/19-15:16:07.231684 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-01 2024/11/19-15:16:07.231789 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00040/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:16:07.661763 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-00 2024/11/19-15:16:07.661848 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00042/att-00/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:16:07.661867 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00042/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.879 sec, real 11.496 (submit 280.685 ms wait 11.207 s), input 126.879 MiB, send shutdown 2024/11/19-15:16:07.669988 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-00 2024/11/19-15:16:07.670202 139851127064320 [dcompact_etcd.cc:1773] job-00042/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:07.762048 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-01 2024/11/19-15:16:07.762151 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00041/att-01/compact.done) = No such file or directory: 2.914 sec on http probe failed: 1, retry 2024/11/19-15:16:07.944389 139851127064320 [dcompact_etcd.cc:1163] job-00042/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.274 sec, input = 126.879 MiB 2024/11/19-15:16:08.637815 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-01 2024/11/19-15:16:08.637892 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00026/att-01/compact.done) = No such file or directory: 13.034 sec on http probe failed: 5, giveup 2024/11/19-15:16:08.637910 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00026/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 223.779 sec, real 13.367 (submit 327.249 ms wait 13.034 s), input 223.779 MiB, send shutdown 2024/11/19-15:16:08.645668 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-01 2024/11/19-15:16:08.645916 139851148035840 [dcompact_etcd.cc:1773] job-00026/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:08.752958 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-01 2024/11/19-15:16:08.753037 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00027/att-01/compact.done) = No such file or directory: 13.030 sec on http probe failed: 5, giveup 2024/11/19-15:16:08.753056 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00027/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.637 sec, real 13.330 (submit 291.125 ms wait 13.030 s), input 255.637 MiB, send shutdown 2024/11/19-15:16:08.758471 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-01 2024/11/19-15:16:08.759587 139851158521600 [dcompact_etcd.cc:1773] job-00027/att-01: CleanFiles 0.001 sec : 0.000 2024/11/19-15:16:08.806580 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-01 2024/11/19-15:16:08.806681 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00028/att-01/compact.done) = No such file or directory: 13.036 sec on http probe failed: 5, giveup 2024/11/19-15:16:08.806701 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00028/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.641 sec, real 13.354 (submit 309.650 ms wait 13.036 s), input 255.641 MiB, send shutdown 2024/11/19-15:16:08.811455 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-01 2024/11/19-15:16:08.811789 139851137550080 [dcompact_etcd.cc:1773] job-00028/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:08.923526 139851148035840 [dcompact_etcd.cc:1163] job-00026/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.278 sec, input = 223.779 MiB 2024/11/19-15:16:09.043287 139851158521600 [dcompact_etcd.cc:1163] job-00027/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.284 sec, input = 255.637 MiB 2024/11/19-15:16:09.098636 139851137550080 [dcompact_etcd.cc:1163] job-00028/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.287 sec, input = 255.641 MiB 2024/11/19-15:16:09.131636 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-01 2024/11/19-15:16:09.131718 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00035/att-01/compact.done) = No such file or directory: 10.905 sec on http probe failed: 5, giveup 2024/11/19-15:16:09.131736 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00035/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 114.819 sec, real 11.192 (submit 276.504 ms wait 10.905 s), input 114.819 MiB, send shutdown 2024/11/19-15:16:09.139723 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-01 2024/11/19-15:16:09.139949 139850978166528 [dcompact_etcd.cc:1773] job-00035/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:09.240674 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-01 2024/11/19-15:16:09.240767 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00040/att-01/compact.done) = No such file or directory: 5.207 sec on http probe failed: 2, retry 2024/11/19-15:16:09.417339 139850978166528 [dcompact_etcd.cc:1163] job-00035/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.277 sec, input = 114.819 MiB 2024/11/19-15:16:09.771205 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-01 2024/11/19-15:16:09.771288 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00041/att-01/compact.done) = No such file or directory: 4.922 sec on http probe failed: 2, retry 2024/11/19-15:16:11.125412 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-01 2024/11/19-15:16:11.125504 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00042/att-01/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:16:11.249357 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-01 2024/11/19-15:16:11.249496 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00040/att-01/compact.done) = No such file or directory: 7.216 sec on http probe failed: 3, retry 2024/11/19-15:16:11.780988 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-01 2024/11/19-15:16:11.781088 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00041/att-01/compact.done) = No such file or directory: 6.931 sec on http probe failed: 3, retry 2024/11/19-15:16:12.297056 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-02 2024/11/19-15:16:12.297156 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00035/att-02/compact.done) = No such file or directory: 2.871 sec on http probe failed: 1, retry 2024/11/19-15:16:13.130759 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-01 2024/11/19-15:16:13.130879 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00042/att-01/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:16:13.258300 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-01 2024/11/19-15:16:13.258377 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00040/att-01/compact.done) = No such file or directory: 9.225 sec on http probe failed: 4, retry 2024/11/19-15:16:13.788147 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-01 2024/11/19-15:16:13.788223 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00041/att-01/compact.done) = No such file or directory: 8.941 sec on http probe failed: 4, retry 2024/11/19-15:16:13.933096 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-02 2024/11/19-15:16:13.933209 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00026/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:16:14.051749 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-02 2024/11/19-15:16:14.051849 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00027/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:16:14.107180 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-02 2024/11/19-15:16:14.107257 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00028/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:16:14.306135 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-02 2024/11/19-15:16:14.306263 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00035/att-02/compact.done) = No such file or directory: 4.880 sec on http probe failed: 2, retry 2024/11/19-15:16:15.140742 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-01 2024/11/19-15:16:15.140841 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00042/att-01/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:16:15.267056 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-01 2024/11/19-15:16:15.267152 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00040/att-01/compact.done) = No such file or directory: 11.233 sec on http probe failed: 5, giveup 2024/11/19-15:16:15.267171 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00040/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.821 sec, real 11.553 (submit 311.309 ms wait 11.233 s), input 127.821 MiB, send shutdown 2024/11/19-15:16:15.274954 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-01 2024/11/19-15:16:15.275176 139851179493120 [dcompact_etcd.cc:1773] job-00040/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:15.553753 139851179493120 [dcompact_etcd.cc:1163] job-00040/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.278 sec, input = 127.821 MiB 2024/11/19-15:16:15.793333 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-01 2024/11/19-15:16:15.793430 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00041/att-01/compact.done) = No such file or directory: 10.948 sec on http probe failed: 5, giveup 2024/11/19-15:16:15.793450 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00041/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 116.546 sec, real 11.202 (submit 248.886 ms wait 10.948 s), input 116.546 MiB, send shutdown 2024/11/19-15:16:15.803118 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-01 2024/11/19-15:16:15.803357 139851169007360 [dcompact_etcd.cc:1773] job-00041/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:15.942937 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-02 2024/11/19-15:16:15.943033 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00026/att-02/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:16:16.060332 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-02 2024/11/19-15:16:16.060420 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00027/att-02/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:16:16.076940 139851169007360 [dcompact_etcd.cc:1163] job-00041/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.273 sec, input = 116.546 MiB 2024/11/19-15:16:16.114793 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-02 2024/11/19-15:16:16.114855 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00028/att-02/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:16:16.309936 139852349699840 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:16:16.315272 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-02 2024/11/19-15:16:16.315362 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00035/att-02/compact.done) = No such file or directory: 6.889 sec on http probe failed: 3, retry 2024/11/19-15:16:17.150812 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-01 2024/11/19-15:16:17.150910 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00042/att-01/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:16:17.951264 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-02 2024/11/19-15:16:17.951356 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00026/att-02/compact.done) = No such file or directory: 9.020 sec on http probe failed: 3, retry 2024/11/19-15:16:18.065468 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-02 2024/11/19-15:16:18.065562 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00027/att-02/compact.done) = No such file or directory: 9.017 sec on http probe failed: 3, retry 2024/11/19-15:16:18.124253 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-02 2024/11/19-15:16:18.124323 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00028/att-02/compact.done) = No such file or directory: 9.016 sec on http probe failed: 3, retry 2024/11/19-15:16:18.325598 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-02 2024/11/19-15:16:18.325688 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00035/att-02/compact.done) = No such file or directory: 8.898 sec on http probe failed: 4, retry 2024/11/19-15:16:18.759409 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-02 2024/11/19-15:16:18.759549 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00040/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:16:18.999819 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-02 2024/11/19-15:16:19.000000 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00041/att-02/compact.done) = No such file or directory: 2.914 sec on http probe failed: 1, retry 2024/11/19-15:16:19.159036 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-01 2024/11/19-15:16:19.159147 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00042/att-01/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:16:19.159171 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00042/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.879 sec, real 11.489 (submit 274.107 ms wait 11.207 s), input 126.879 MiB, send shutdown 2024/11/19-15:16:19.163740 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-01 2024/11/19-15:16:19.229328 139851127064320 [dcompact_etcd.cc:1773] job-00042/att-01: CleanFiles 0.066 sec : 0.000 2024/11/19-15:16:19.530389 139851127064320 [dcompact_etcd.cc:1163] job-00042/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.301 sec, input = 126.879 MiB 2024/11/19-15:16:19.960513 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-02 2024/11/19-15:16:19.960643 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00026/att-02/compact.done) = No such file or directory: 11.028 sec on http probe failed: 4, retry 2024/11/19-15:16:20.073235 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-02 2024/11/19-15:16:20.073329 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00027/att-02/compact.done) = No such file or directory: 11.022 sec on http probe failed: 4, retry 2024/11/19-15:16:20.133630 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-02 2024/11/19-15:16:20.133700 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00028/att-02/compact.done) = No such file or directory: 11.026 sec on http probe failed: 4, retry 2024/11/19-15:16:20.334658 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-02 2024/11/19-15:16:20.334747 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00035/att-02/compact.done) = No such file or directory: 10.908 sec on http probe failed: 5, giveup 2024/11/19-15:16:20.334766 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00035/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 114.819 sec, real 11.195 (submit 277.293 ms wait 10.908 s), input 114.819 MiB, send shutdown 2024/11/19-15:16:20.342933 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00035/att-02 2024/11/19-15:16:20.428664 139850978166528 [dcompact_etcd.cc:1773] job-00035/att-02: CleanFiles 0.086 sec : 0.000 2024/11/19-15:16:20.602089 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00035) = success 2024/11/19-15:16:20.602128 139850978166528 [dcompact_etcd.cc:1773] job-00035/att-03: CleanFiles 0.173 sec : 0.000 2024/11/19-15:16:20.602174 139850978166528 (Original Log Time 2024/11/19-15:15:46.740150) EVENT_LOG_v1 {"time_micros": 1732000546740144, "job": 35, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4355, 4358, 4360, 4362], "score": 4.14809, "input_data_size": 120999680, "oldest_snapshot_seqno": 109450670} 2024/11/19-15:16:20.602186 139850978166528 [compaction_job.cc:2437] [default] [JOB 35] Compacting 4@1 files to L2, score 4.15, subcompactions 1 : 1 2024/11/19-15:16:20.602202 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 33 Base level 1, inputs: [4355(31MB) 4358(31MB) 4360(31MB) 4362(19MB)] 2024/11/19-15:16:20.602617 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004433.sst) /storage/mytopling/..//tmp/Topling-82038-Vud4fd 2024/11/19-15:16:20.669726 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004433.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:16:20.678275 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004433.sst): index pass time = 0.01's, 141.502'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 100331 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:16:20.742368 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004433.sst): first pass time = 0.14's, 480.283'MB/sec 2024/11/19-15:16:20.742424 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004433.sst): old prealloc_size = 73819750, real_size = 65222896 2024/11/19-15:16:20.742495 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004433.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 100331 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0650 GB avg-val = 644.88 avg-zval = 648.13 2024/11/19-15:16:20.742542 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004433.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.073 sec, Key+Value = 0.067 GB 2024/11/19-15:16:20.768559 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-02 2024/11/19-15:16:20.768651 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00040/att-02/compact.done) = No such file or directory: 5.206 sec on http probe failed: 2, retry 2024/11/19-15:16:20.809643 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004433.sst): second pass time = 0.00's,162975284.635'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 970.109'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2487.709'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.558'MB/sec write SST data time = 0.02's, 2939.392'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 162975284.635'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 100331 avg-key = 24.00 avg-zkey = 1.70 avg-val = 644.88 avg-zval = 648.13 usrkeys = 100331 avg-key = 16.00 avg-zkey = 1.70 avg-val = 644.88 avg-zval = 648.13 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7942 UnZipSize{ index = 0.0016 GB value = 0.0647 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0002 GB value = 0.0650 GB dict = 0.00 MB all = 0.0652 GB } UnZip/Zip{ index = 9.4367 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9715 } ---------------------------- total value len = 1.550601 GB avg = 0.480 KB (by entry num) total key len = 0.077428 GB avg = 0.024 KB total ukey len = 0.051580 GB avg = 0.016 KB total ukey num = 0.003231011 Billion total entry num = 0.003231011 Billion write speed all = 18.913197674 MB/sec (with seq num) write speed all = 18.612914349 MB/sec (without seq num) 2024/11/19-15:16:20.813525 139850978166528 [compaction_job.cc:1948] [default] [JOB 35] Generated table #4433: 100331 keys, 65199662 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:20.813582 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000580813558, "cf_name": "default", "job": 35, "event": "table_file_creation", "file_number": 4433, "file_size": 65199662, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 65027488, "index_size": 170112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2407944, "raw_average_key_size": 24, "raw_value_size": 64701188, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 100331, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000580, "oldest_key_time": 0, "file_creation_time": 1732000580, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4433, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:20.830054 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004434.sst) /storage/mytopling/..//tmp/Topling-82038-s7ibwy 2024/11/19-15:16:20.878772 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004434.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.053 GB 2024/11/19-15:16:20.885584 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004434.sst): index pass time = 0.01's, 141.199'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 79669 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0001 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:16:20.935511 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004434.sst): first pass time = 0.11's, 505.382'MB/sec 2024/11/19-15:16:20.935571 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004434.sst): old prealloc_size = 73819750, real_size = 51789952 2024/11/19-15:16:20.935634 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004434.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 79669 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0001 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0514 GB zip-val = 0.0516 GB avg-val = 644.86 avg-zval = 648.11 2024/11/19-15:16:20.935682 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004434.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.057 sec, Key+Value = 0.053 GB 2024/11/19-15:16:20.989484 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004434.sst): second pass time = 0.00's,145952468.750'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 960.432'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2302.385'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.564'MB/sec write SST data time = 0.02's, 2789.858'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.051'GB zip my value throughput = 145952468.750'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 79669 avg-key = 24.00 avg-zkey = 1.70 avg-val = 644.86 avg-zval = 648.11 usrkeys = 79669 avg-key = 16.00 avg-zkey = 1.70 avg-val = 644.86 avg-zval = 648.11 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6308 UnZipSize{ index = 0.0013 GB value = 0.0514 GB dict = 0.00 MB all = 0.0533 GB } __ZipSize{ index = 0.0001 GB value = 0.0516 GB dict = 0.00 MB all = 0.0518 GB } UnZip/Zip{ index = 9.4127 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1062 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 1.601977 GB avg = 0.484 KB (by entry num) total key len = 0.079340 GB avg = 0.024 KB total ukey len = 0.052854 GB avg = 0.016 KB total ukey num = 0.003310680 Billion total entry num = 0.003310680 Billion write speed all = 19.491527040 MB/sec (with seq num) write speed all = 19.184480933 MB/sec (without seq num) 2024/11/19-15:16:20.992500 139850978166528 [compaction_job.cc:1948] [default] [JOB 35] Generated table #4434: 79669 keys, 51771757 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:20.992564 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000580992528, "cf_name": "default", "job": 35, "event": "table_file_creation", "file_number": 4434, "file_size": 51771757, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 51634416, "index_size": 135424, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1912056, "raw_average_key_size": 24, "raw_value_size": 51375269, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 79669, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000580, "oldest_key_time": 0, "file_creation_time": 1732000580, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4434, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:21.005464 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-02 2024/11/19-15:16:21.005561 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00041/att-02/compact.done) = No such file or directory: 4.923 sec on http probe failed: 2, retry 2024/11/19-15:16:21.006205 139850978166528 [compaction_job.cc:703] job-00035: subcompact[0], size: 0.116971 G, files: /storage/mytopling/.rocksdb [004433,004434] 2024/11/19-15:16:21.007102 139850978166528 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:16:21.007204 139850978166528 (Original Log Time 2024/11/19-15:16:20.602262) EVENT_LOG_v1 {"time_micros": 1732000580602225, "job": 35, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4355, 4358, 4360, 4362], "score": 4.14809, "input_data_size": 120999680, "oldest_snapshot_seqno": 109450670} 2024/11/19-15:16:21.007210 139850978166528 (Original Log Time 2024/11/19-15:16:21.006678) [compaction_job.cc:2022] [default] [JOB 35] Compacted 4@1 files to L2 => 116971419 bytes 2024/11/19-15:16:21.007213 139850978166528 (Original Log Time 2024/11/19-15:16:21.007071) [compaction_job.cc:1224] [default] compacted to: files[4 63 77 75 0 0 0] max score 3.15 (120 files need compaction), MB/sec: 299.5 rd, 289.6 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(115.4, 0.0 +0.0 blob) out(111.6 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 180000, records dropped: 0 output_compression: Snappy 2024/11/19-15:16:21.007232 139850978166528 (Original Log Time 2024/11/19-15:16:21.007096) EVENT_LOG_v1 {"time_micros": 1732000581007084, "cf": "default", "job": 35, "event": "compaction_finished", "compaction_time_micros": 403956, "compaction_time_cpu_micros": 400135, "output_level": 2, "num_output_files": 2, "total_output_size": 116971419, "num_input_records": 180000, "num_output_records": 180000, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 63, 77, 75, 0, 0, 0]} 2024/11/19-15:16:21.007308 139850978166528 [compaction_job.cc:2437] [default] [JOB 44] Compacting 4@1 files to L2, score 3.15, subcompactions 1 : 1 2024/11/19-15:16:21.007316 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 42 Base level 1, inputs: [4402(31MB) 4403(31MB) 4405(31MB) 4407(31MB)] 2024/11/19-15:16:21.309422 139850978166528 [dcompact_etcd.cc:1163] job-00044/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.302 sec, input = 126.907 MiB 2024/11/19-15:16:21.969450 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-02 2024/11/19-15:16:21.969540 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00026/att-02/compact.done) = No such file or directory: 13.037 sec on http probe failed: 5, giveup 2024/11/19-15:16:21.969559 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00026/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 223.779 sec, real 13.324 (submit 277.503 ms wait 13.037 s), input 223.779 MiB, send shutdown 2024/11/19-15:16:21.975227 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00026/att-02 2024/11/19-15:16:21.975472 139851148035840 [dcompact_etcd.cc:1773] job-00026/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:21.975724 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00026) = success 2024/11/19-15:16:21.975730 139851148035840 [dcompact_etcd.cc:1773] job-00026/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:21.975749 139851148035840 (Original Log Time 2024/11/19-15:15:41.921603) EVENT_LOG_v1 {"time_micros": 1732000541921596, "job": 26, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4293, 4304, 4312, 4305], "score": 2.43141, "input_data_size": 194654929, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:21.975756 139851148035840 [compaction_job.cc:2437] [default] [JOB 26] Compacting 4@2 files to L3, score 2.43, subcompactions 1 : 1 2024/11/19-15:16:21.975763 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 24 Base level 2, inputs: [4293(26MB) 4304(52MB) 4312(52MB) 4305(52MB)] 2024/11/19-15:16:22.078099 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-02 2024/11/19-15:16:22.078188 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00027/att-02/compact.done) = No such file or directory: 13.030 sec on http probe failed: 5, giveup 2024/11/19-15:16:22.078207 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00027/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.637 sec, real 13.319 (submit 283.604 ms wait 13.030 s), input 255.637 MiB, send shutdown 2024/11/19-15:16:22.082692 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00027/att-02 2024/11/19-15:16:22.082942 139851158521600 [dcompact_etcd.cc:1773] job-00027/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:22.083179 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00027) = success 2024/11/19-15:16:22.083185 139851158521600 [dcompact_etcd.cc:1773] job-00027/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:22.083203 139851158521600 (Original Log Time 2024/11/19-15:15:42.055854) EVENT_LOG_v1 {"time_micros": 1732000542055847, "job": 27, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4311, 4308, 4314, 4307], "score": 2.3683, "input_data_size": 221714180, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:22.083210 139851158521600 [compaction_job.cc:2437] [default] [JOB 27] Compacting 4@2 files to L3, score 2.37, subcompactions 1 : 1 2024/11/19-15:16:22.083218 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 25 Base level 2, inputs: [4311(52MB) 4308(52MB) 4314(52MB) 4307(52MB)] 2024/11/19-15:16:22.140946 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-02 2024/11/19-15:16:22.141041 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00028/att-02/compact.done) = No such file or directory: 13.035 sec on http probe failed: 5, giveup 2024/11/19-15:16:22.141065 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00028/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.641 sec, real 13.329 (submit 286.750 ms wait 13.035 s), input 255.641 MiB, send shutdown 2024/11/19-15:16:22.149980 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00028/att-02 2024/11/19-15:16:22.150259 139851137550080 [dcompact_etcd.cc:1773] job-00028/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:22.150533 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00028) = success 2024/11/19-15:16:22.150542 139851137550080 [dcompact_etcd.cc:1773] job-00028/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:22.150567 139851137550080 (Original Log Time 2024/11/19-15:15:42.093632) EVENT_LOG_v1 {"time_micros": 1732000542093625, "job": 28, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4315, 4331, 4338, 4322], "score": 2.28443, "input_data_size": 221713668, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:22.150578 139851137550080 [compaction_job.cc:2437] [default] [JOB 28] Compacting 4@2 files to L3, score 2.28, subcompactions 1 : 1 2024/11/19-15:16:22.150589 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 26 Base level 2, inputs: [4315(52MB) 4331(52MB) 4338(52MB) 4322(52MB)] 2024/11/19-15:16:22.163721 139851148035840 [compaction_job.cc:703] job-00026: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:16:22.164310 139851148035840 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:16:22.164424 139851148035840 (Original Log Time 2024/11/19-15:16:21.975791) EVENT_LOG_v1 {"time_micros": 1732000581975775, "job": 26, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4293, 4304, 4312, 4305], "score": 2.43141, "input_data_size": 194654929, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:22.164433 139851148035840 (Original Log Time 2024/11/19-15:16:22.163793) [compaction_job.cc:2022] [default] [JOB 26] Compacted 4@2 files to L3 => 0 bytes 2024/11/19-15:16:22.164438 139851148035840 (Original Log Time 2024/11/19-15:16:22.164274) [compaction_job.cc:1224] [default] compacted to: files[4 63 73 75 0 0 0] max score 2.90 (116 files need compaction), MB/sec: 1035.7 rd, 0.0 wr, level 3, files in(4, 0) out(0 +0 blob) MB in(185.6, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 1620475, records dropped: 1620475 output_compression: Snappy 2024/11/19-15:16:22.164460 139851148035840 (Original Log Time 2024/11/19-15:16:22.164304) EVENT_LOG_v1 {"time_micros": 1732000582164289, "cf": "default", "job": 26, "event": "compaction_finished", "compaction_time_micros": 187946, "compaction_time_cpu_micros": 187291, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 1620475, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 63, 73, 75, 0, 0, 0]} 2024/11/19-15:16:22.170563 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004210.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.170595 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000582170591, "job": 26, "event": "table_file_deletion", "file_number": 4210} 2024/11/19-15:16:22.176599 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004211.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.176626 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000582176623, "job": 26, "event": "table_file_deletion", "file_number": 4211} 2024/11/19-15:16:22.182665 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004213.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.182691 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000582182688, "job": 26, "event": "table_file_deletion", "file_number": 4213} 2024/11/19-15:16:22.188723 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004214.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.188747 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000582188745, "job": 26, "event": "table_file_deletion", "file_number": 4214} 2024/11/19-15:16:22.188968 139851148035840 [compaction_job.cc:2437] [default] [JOB 45] Compacting 4@1 files to L2, score 2.90, subcompactions 1 : 1 2024/11/19-15:16:22.188982 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 43 Base level 1, inputs: [4409(31MB) 4410(31MB) 4412(31MB) 4414(31MB)] 2024/11/19-15:16:22.286421 139851158521600 [compaction_job.cc:703] job-00027: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:16:22.286909 139851158521600 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:16:22.287015 139851158521600 (Original Log Time 2024/11/19-15:16:22.083248) EVENT_LOG_v1 {"time_micros": 1732000582083230, "job": 27, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4311, 4308, 4314, 4307], "score": 2.3683, "input_data_size": 221714180, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:22.287022 139851158521600 (Original Log Time 2024/11/19-15:16:22.286512) [compaction_job.cc:2022] [default] [JOB 27] Compacted 4@2 files to L3 => 0 bytes 2024/11/19-15:16:22.287025 139851158521600 (Original Log Time 2024/11/19-15:16:22.286876) [compaction_job.cc:1224] [default] compacted to: files[4 63 69 75 0 0 0] max score 2.65 (112 files need compaction), MB/sec: 1091.1 rd, 0.0 wr, level 3, files in(4, 0) out(0 +0 blob) MB in(211.4, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 1851075, records dropped: 1851075 output_compression: Snappy 2024/11/19-15:16:22.287041 139851158521600 (Original Log Time 2024/11/19-15:16:22.286902) EVENT_LOG_v1 {"time_micros": 1732000582286890, "cf": "default", "job": 27, "event": "compaction_finished", "compaction_time_micros": 203198, "compaction_time_cpu_micros": 202601, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 1851075, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 63, 69, 75, 0, 0, 0]} 2024/11/19-15:16:22.291002 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004204.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.291033 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000582291029, "job": 27, "event": "table_file_deletion", "file_number": 4204} 2024/11/19-15:16:22.295026 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004205.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.295049 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000582295046, "job": 27, "event": "table_file_deletion", "file_number": 4205} 2024/11/19-15:16:22.299158 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004207.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.299178 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000582299176, "job": 27, "event": "table_file_deletion", "file_number": 4207} 2024/11/19-15:16:22.303175 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004208.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.303195 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000582303193, "job": 27, "event": "table_file_deletion", "file_number": 4208} 2024/11/19-15:16:22.303373 139851158521600 [compaction_job.cc:2437] [default] [JOB 46] Compacting 4@1 files to L2, score 2.65, subcompactions 1 : 1 2024/11/19-15:16:22.303382 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 44 Base level 1, inputs: [4416(31MB) 4418(31MB) 4420(31MB) 4421(31MB)] 2024/11/19-15:16:22.386347 139851137550080 [compaction_job.cc:703] job-00028: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:16:22.386800 139851137550080 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:16:22.386955 139851137550080 (Original Log Time 2024/11/19-15:16:22.150631) EVENT_LOG_v1 {"time_micros": 1732000582150607, "job": 28, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4315, 4331, 4338, 4322], "score": 2.28443, "input_data_size": 221713668, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:22.386962 139851137550080 (Original Log Time 2024/11/19-15:16:22.386426) [compaction_job.cc:2022] [default] [JOB 28] Compacted 4@2 files to L3 => 0 bytes 2024/11/19-15:16:22.386965 139851137550080 (Original Log Time 2024/11/19-15:16:22.386768) [compaction_job.cc:1224] [default] compacted to: files[4 63 65 75 0 0 0] max score 2.40 (108 files need compaction), MB/sec: 940.5 rd, 0.0 wr, level 3, files in(4, 0) out(0 +0 blob) MB in(211.4, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 1851266, records dropped: 1851266 output_compression: Snappy 2024/11/19-15:16:22.386970 139851137550080 (Original Log Time 2024/11/19-15:16:22.386793) EVENT_LOG_v1 {"time_micros": 1732000582386782, "cf": "default", "job": 28, "event": "compaction_finished", "compaction_time_micros": 235739, "compaction_time_cpu_micros": 235043, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 1851266, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 63, 65, 75, 0, 0, 0]} 2024/11/19-15:16:22.391033 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004215.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.391061 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582391058, "job": 28, "event": "table_file_deletion", "file_number": 4215} 2024/11/19-15:16:22.394979 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004217.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.395002 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582394999, "job": 28, "event": "table_file_deletion", "file_number": 4217} 2024/11/19-15:16:22.398988 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004218.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.399008 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582399006, "job": 28, "event": "table_file_deletion", "file_number": 4218} 2024/11/19-15:16:22.403015 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004220.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.403036 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582403034, "job": 28, "event": "table_file_deletion", "file_number": 4220} 2024/11/19-15:16:22.407409 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004221.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.407428 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582407426, "job": 28, "event": "table_file_deletion", "file_number": 4221} 2024/11/19-15:16:22.411852 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004222.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.411870 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582411868, "job": 28, "event": "table_file_deletion", "file_number": 4222} 2024/11/19-15:16:22.415990 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004224.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.416009 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582416007, "job": 28, "event": "table_file_deletion", "file_number": 4224} 2024/11/19-15:16:22.420129 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004226.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.420149 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582420147, "job": 28, "event": "table_file_deletion", "file_number": 4226} 2024/11/19-15:16:22.424573 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004228.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.424594 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582424592, "job": 28, "event": "table_file_deletion", "file_number": 4228} 2024/11/19-15:16:22.428578 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004230.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.428598 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582428596, "job": 28, "event": "table_file_deletion", "file_number": 4230} 2024/11/19-15:16:22.432655 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004232.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.432674 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582432672, "job": 28, "event": "table_file_deletion", "file_number": 4232} 2024/11/19-15:16:22.436715 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004233.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.436734 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582436732, "job": 28, "event": "table_file_deletion", "file_number": 4233} 2024/11/19-15:16:22.440731 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004234.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.440749 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582440747, "job": 28, "event": "table_file_deletion", "file_number": 4234} 2024/11/19-15:16:22.444804 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004236.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.444822 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582444820, "job": 28, "event": "table_file_deletion", "file_number": 4236} 2024/11/19-15:16:22.448860 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004237.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.448879 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582448877, "job": 28, "event": "table_file_deletion", "file_number": 4237} 2024/11/19-15:16:22.452874 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004238.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.452893 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582452891, "job": 28, "event": "table_file_deletion", "file_number": 4238} 2024/11/19-15:16:22.456981 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004240.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.457000 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582456998, "job": 28, "event": "table_file_deletion", "file_number": 4240} 2024/11/19-15:16:22.460963 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004241.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.460987 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582460985, "job": 28, "event": "table_file_deletion", "file_number": 4241} 2024/11/19-15:16:22.464929 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004242.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.464949 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582464947, "job": 28, "event": "table_file_deletion", "file_number": 4242} 2024/11/19-15:16:22.469011 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004244.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.469030 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582469028, "job": 28, "event": "table_file_deletion", "file_number": 4244} 2024/11/19-15:16:22.473177 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004245.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.473197 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582473195, "job": 28, "event": "table_file_deletion", "file_number": 4245} 2024/11/19-15:16:22.478169 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004246.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.478190 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582478187, "job": 28, "event": "table_file_deletion", "file_number": 4246} 2024/11/19-15:16:22.482564 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004247.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.482583 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582482581, "job": 28, "event": "table_file_deletion", "file_number": 4247} 2024/11/19-15:16:22.487665 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004248.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.487683 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582487681, "job": 28, "event": "table_file_deletion", "file_number": 4248} 2024/11/19-15:16:22.492272 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004249.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.492290 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582492288, "job": 28, "event": "table_file_deletion", "file_number": 4249} 2024/11/19-15:16:22.496972 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004250.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.496990 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582496988, "job": 28, "event": "table_file_deletion", "file_number": 4250} 2024/11/19-15:16:22.501010 139851148035840 [dcompact_etcd.cc:1163] job-00045/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.311 sec, input = 126.825 MiB 2024/11/19-15:16:22.501220 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004251.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.501240 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582501238, "job": 28, "event": "table_file_deletion", "file_number": 4251} 2024/11/19-15:16:22.505802 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004252.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.505822 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582505820, "job": 28, "event": "table_file_deletion", "file_number": 4252} 2024/11/19-15:16:22.510320 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004253.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.510339 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582510337, "job": 28, "event": "table_file_deletion", "file_number": 4253} 2024/11/19-15:16:22.514506 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004254.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.514531 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582514530, "job": 28, "event": "table_file_deletion", "file_number": 4254} 2024/11/19-15:16:22.518704 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004255.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.518723 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582518721, "job": 28, "event": "table_file_deletion", "file_number": 4255} 2024/11/19-15:16:22.522895 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004256.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.522913 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582522912, "job": 28, "event": "table_file_deletion", "file_number": 4256} 2024/11/19-15:16:22.527044 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004257.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.527062 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582527060, "job": 28, "event": "table_file_deletion", "file_number": 4257} 2024/11/19-15:16:22.531689 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004258.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.531707 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582531706, "job": 28, "event": "table_file_deletion", "file_number": 4258} 2024/11/19-15:16:22.535887 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004259.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.535905 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582535904, "job": 28, "event": "table_file_deletion", "file_number": 4259} 2024/11/19-15:16:22.540257 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004260.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.540275 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582540273, "job": 28, "event": "table_file_deletion", "file_number": 4260} 2024/11/19-15:16:22.544397 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004261.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.544416 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582544415, "job": 28, "event": "table_file_deletion", "file_number": 4261} 2024/11/19-15:16:22.548896 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004262.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.548914 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582548912, "job": 28, "event": "table_file_deletion", "file_number": 4262} 2024/11/19-15:16:22.553080 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004263.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.553098 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582553097, "job": 28, "event": "table_file_deletion", "file_number": 4263} 2024/11/19-15:16:22.557222 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004264.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.557241 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582557239, "job": 28, "event": "table_file_deletion", "file_number": 4264} 2024/11/19-15:16:22.557872 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004265.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.557887 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582557885, "job": 28, "event": "table_file_deletion", "file_number": 4265} 2024/11/19-15:16:22.611844 139851158521600 [dcompact_etcd.cc:1163] job-00046/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.308 sec, input = 126.836 MiB 2024/11/19-15:16:22.626501 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004269.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.626544 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582626539, "job": 28, "event": "table_file_deletion", "file_number": 4269} 2024/11/19-15:16:22.696167 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004276.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.696207 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582696204, "job": 28, "event": "table_file_deletion", "file_number": 4276} 2024/11/19-15:16:22.709256 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004284.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.709290 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582709287, "job": 28, "event": "table_file_deletion", "file_number": 4284} 2024/11/19-15:16:22.711182 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-02 2024/11/19-15:16:22.711263 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00042/att-02/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:16:22.711776 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004290.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.711799 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582711797, "job": 28, "event": "table_file_deletion", "file_number": 4290} 2024/11/19-15:16:22.713897 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004292.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.713917 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582713915, "job": 28, "event": "table_file_deletion", "file_number": 4292} 2024/11/19-15:16:22.714561 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004302.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.714575 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582714574, "job": 28, "event": "table_file_deletion", "file_number": 4302} 2024/11/19-15:16:22.723307 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004303.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.723327 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582723325, "job": 28, "event": "table_file_deletion", "file_number": 4303} 2024/11/19-15:16:22.731436 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004306.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.731460 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582731457, "job": 28, "event": "table_file_deletion", "file_number": 4306} 2024/11/19-15:16:22.741718 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004309.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.741742 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582741740, "job": 28, "event": "table_file_deletion", "file_number": 4309} 2024/11/19-15:16:22.749511 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004310.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.749539 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582749537, "job": 28, "event": "table_file_deletion", "file_number": 4310} 2024/11/19-15:16:22.750440 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004323.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.750457 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582750455, "job": 28, "event": "table_file_deletion", "file_number": 4323} 2024/11/19-15:16:22.750679 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004351.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.750688 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582750687, "job": 28, "event": "table_file_deletion", "file_number": 4351} 2024/11/19-15:16:22.751242 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004352.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.751255 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582751254, "job": 28, "event": "table_file_deletion", "file_number": 4352} 2024/11/19-15:16:22.751381 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004354.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.751389 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582751388, "job": 28, "event": "table_file_deletion", "file_number": 4354} 2024/11/19-15:16:22.751782 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004367.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.751793 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582751792, "job": 28, "event": "table_file_deletion", "file_number": 4367} 2024/11/19-15:16:22.752489 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004368.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.752503 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582752502, "job": 28, "event": "table_file_deletion", "file_number": 4368} 2024/11/19-15:16:22.752723 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004425.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:22.752732 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000582752731, "job": 28, "event": "table_file_deletion", "file_number": 4425} 2024/11/19-15:16:22.752988 139851137550080 [compaction_job.cc:2437] [default] [JOB 47] Compacting 4@1 files to L2, score 2.40, subcompactions 1 : 1 2024/11/19-15:16:22.752999 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 45 Base level 1, inputs: [4422(31MB) 4423(26MB) 4353(31MB) 4357(31MB)] 2024/11/19-15:16:22.773826 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-02 2024/11/19-15:16:22.773882 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00040/att-02/compact.done) = No such file or directory: 7.215 sec on http probe failed: 3, retry 2024/11/19-15:16:23.012038 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-02 2024/11/19-15:16:23.012134 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00041/att-02/compact.done) = No such file or directory: 6.929 sec on http probe failed: 3, retry 2024/11/19-15:16:23.177975 139851137550080 [dcompact_etcd.cc:1163] job-00047/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.425 sec, input = 121.888 MiB 2024/11/19-15:16:24.490615 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-00 2024/11/19-15:16:24.490713 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00044/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:16:24.720713 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-02 2024/11/19-15:16:24.720813 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00042/att-02/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:16:24.782051 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-02 2024/11/19-15:16:24.782172 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00040/att-02/compact.done) = No such file or directory: 9.220 sec on http probe failed: 4, retry 2024/11/19-15:16:25.021088 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-02 2024/11/19-15:16:25.021194 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00041/att-02/compact.done) = No such file or directory: 8.935 sec on http probe failed: 4, retry 2024/11/19-15:16:25.676904 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4435. Immutable memtables: 0. 2024/11/19-15:16:25.676974 139852362282752 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:16:25.677238 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 48] Syncing log #4431 2024/11/19-15:16:25.677334 139850967680768 (Original Log Time 2024/11/19-15:16:25.677021) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:16:25.677344 139850967680768 [flush_job.cc:879] [default] [JOB 48] Flushing memtable with next log file: 4435 2024/11/19-15:16:25.677365 139850967680768 EVENT_LOG_v1 {"time_micros": 1732000585677360, "job": 48, "event": "flush_started", "num_memtables": 1, "num_entries": 1661846, "num_deletes": 0, "total_data_size": 474307866, "memory_usage": 537394568, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:16:25.677369 139850967680768 [flush_job.cc:910] [default] [JOB 48] Level-0 flush table #4436: started 2024/11/19-15:16:25.678073 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000004.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004436.sst in ConvertToSST, mem_size = 512.499 M, time = 694.311 us 2024/11/19-15:16:25.678290 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-00 2024/11/19-15:16:25.678367 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00045/att-00/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:16:25.793164 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-00 2024/11/19-15:16:25.793305 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00046/att-00/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:16:25.812780 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004436.sst): fsize = 512.501 M, time(ms): open: 0.036, seek: 0.026, finish: 0.039, meta: 0.012, Flush: 0.012, sync: 134.557, close: 0.009, all: 134.691 2024/11/19-15:16:25.814024 139850967680768 [flush_job.cc:1083] [default] [JOB 48] Flush lasted 136689 microseconds, and 112168 cpu microseconds. 2024/11/19-15:16:25.834483 139850967680768 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:16:25.834583 139850967680768 (Original Log Time 2024/11/19-15:16:25.812821) [flush_job.cc:1029] [default] [JOB 48] Level-0 flush table #4436: 537395994 bytes OK (needs compaction) 2024/11/19-15:16:25.834589 139850967680768 (Original Log Time 2024/11/19-15:16:25.814041) [memtable_list.cc:560] [default] Level-0 commit table #4436 started 2024/11/19-15:16:25.834592 139850967680768 (Original Log Time 2024/11/19-15:16:25.834440) [memtable_list.cc:759] [default] Level-0 commit table #4436: memtable #1 done 2024/11/19-15:16:25.834595 139850967680768 (Original Log Time 2024/11/19-15:16:25.834477) EVENT_LOG_v1 {"time_micros": 1732000585834468, "job": 48, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [5, 63, 65, 75, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:16:25.834598 139850967680768 (Original Log Time 2024/11/19-15:16:25.834557) [db_impl_compaction_flush.cc:310] [default] Level summary: files[5 63 65 75 0 0 0] max score 2.26 (105 files need compaction) 2024/11/19-15:16:26.233427 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-00 2024/11/19-15:16:26.233545 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00047/att-00/compact.done) = No such file or directory: 3.047 sec on http probe failed: 1, retry 2024/11/19-15:16:26.499865 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-00 2024/11/19-15:16:26.499953 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00044/att-00/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:16:26.730217 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-02 2024/11/19-15:16:26.730292 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00042/att-02/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:16:26.790538 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-02 2024/11/19-15:16:26.790601 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00040/att-02/compact.done) = No such file or directory: 11.229 sec on http probe failed: 5, giveup 2024/11/19-15:16:26.790620 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00040/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.821 sec, real 11.515 (submit 278.494 ms wait 11.229 s), input 127.821 MiB, send shutdown 2024/11/19-15:16:26.795159 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00040/att-02 2024/11/19-15:16:26.795380 139851179493120 [dcompact_etcd.cc:1773] job-00040/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:26.795636 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00040) = success 2024/11/19-15:16:26.795642 139851179493120 [dcompact_etcd.cc:1773] job-00040/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:26.795661 139851179493120 (Original Log Time 2024/11/19-15:15:52.167560) EVENT_LOG_v1 {"time_micros": 1732000552167554, "job": 40, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4356, 4363, 4370, 4373], "score": 3.88095, "input_data_size": 133300300, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:26.795668 139851179493120 [compaction_job.cc:2437] [default] [JOB 40] Compacting 4@1 files to L2, score 3.88, subcompactions 1 : 1 2024/11/19-15:16:26.795676 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 38 Base level 1, inputs: [4356(31MB) 4363(31MB) 4370(31MB) 4373(31MB)] 2024/11/19-15:16:26.795917 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004437.sst) /storage/mytopling/..//tmp/Topling-82038-zAypLY 2024/11/19-15:16:26.941855 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004437.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:16:26.975557 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004437.sst): index pass time = 0.03's, 220.221'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463349 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:16:27.030505 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-02 2024/11/19-15:16:27.030615 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00041/att-02/compact.done) = No such file or directory: 10.944 sec on http probe failed: 5, giveup 2024/11/19-15:16:27.030641 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00041/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 116.546 sec, real 11.227 (submit 273.486 ms wait 10.944 s), input 116.546 MiB, send shutdown 2024/11/19-15:16:27.038647 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00041/att-02 2024/11/19-15:16:27.038939 139851169007360 [dcompact_etcd.cc:1773] job-00041/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:27.039244 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00041) = success 2024/11/19-15:16:27.039255 139851169007360 [dcompact_etcd.cc:1773] job-00041/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:27.039280 139851169007360 (Original Log Time 2024/11/19-15:15:53.317465) EVENT_LOG_v1 {"time_micros": 1732000553317458, "job": 41, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4378, 4382, 4387, 4391], "score": 3.6313, "input_data_size": 121592204, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:27.039291 139851169007360 [compaction_job.cc:2437] [default] [JOB 41] Compacting 4@1 files to L2, score 3.63, subcompactions 1 : 1 2024/11/19-15:16:27.039302 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 39 Base level 1, inputs: [4378(31MB) 4382(31MB) 4387(31MB) 4391(20MB)] 2024/11/19-15:16:27.039615 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004438.sst) /storage/mytopling/..//tmp/Topling-82038-QwFkpa 2024/11/19-15:16:27.046045 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004437.sst): first pass time = 0.25's, 268.324'MB/sec 2024/11/19-15:16:27.046100 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004437.sst): old prealloc_size = 73819750, real_size = 55605408 2024/11/19-15:16:27.046353 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004437.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463349 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.83 avg-zval = 117.87 2024/11/19-15:16:27.046422 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004437.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.105 sec, Key+Value = 0.067 GB 2024/11/19-15:16:27.146125 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004437.sst): second pass time = 0.00's,124735341.014'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 554.850'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3468.667'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 2.394'MB/sec write SST data time = 0.05's, 1079.932'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 124735341.014'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463349 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.83 avg-zval = 117.87 usrkeys = 463349 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.83 avg-zval = 117.87 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6616 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5730 value = 0.9913 dict = 0.00 all = 1.2093 } Zip/UnZip{ index = 0.0946 value = 1.0088 dict = 0.00 all = 0.8269 } ---------------------------- total value len = 1.656112 GB avg = 0.439 KB (by entry num) total key len = 0.092314 GB avg = 0.024 KB total ukey len = 0.062121 GB avg = 0.016 KB total ukey num = 0.003774029 Billion total entry num = 0.003774029 Billion write speed all = 18.919182931 MB/sec (with seq num) write speed all = 18.592481904 MB/sec (without seq num) 2024/11/19-15:16:27.149871 139851179493120 [compaction_job.cc:1948] [default] [JOB 40] Generated table #4437: 463349 keys, 55494833 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:27.149931 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000587149899, "cf_name": "default", "job": 40, "event": "table_file_creation", "file_number": 4437, "file_size": 55494833, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54612880, "index_size": 876480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12973772, "raw_average_key_size": 28, "raw_value_size": 54135138, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463349, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000586, "oldest_key_time": 0, "file_creation_time": 1732000586, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4437, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:27.164847 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004439.sst) /storage/mytopling/..//tmp/Topling-82038-zc3CuJ 2024/11/19-15:16:27.202293 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004438.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:16:27.237560 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004438.sst): index pass time = 0.04's, 210.490'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463513 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:16:27.305859 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004438.sst): first pass time = 0.27's, 252.076'MB/sec 2024/11/19-15:16:27.305914 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004438.sst): old prealloc_size = 73819750, real_size = 55619168 2024/11/19-15:16:27.306157 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004438.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463513 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.78 avg-zval = 117.85 2024/11/19-15:16:27.306218 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004438.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.104 sec, Key+Value = 0.067 GB 2024/11/19-15:16:27.317117 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004439.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.001 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:16:27.351836 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004439.sst): index pass time = 0.03's, 213.160'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462168 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.92 2024/11/19-15:16:27.395364 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004438.sst): second pass time = 0.00's,147094135.870'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 620.637'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3808.134'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 2.554'MB/sec write SST data time = 0.04's, 1266.474'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 147094135.870'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463513 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 117.85 usrkeys = 463513 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6615 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5728 value = 0.9909 dict = 0.00 all = 1.2090 } Zip/UnZip{ index = 0.0946 value = 1.0092 dict = 0.00 all = 0.8271 } ---------------------------- total value len = 1.710242 GB avg = 0.404 KB (by entry num) total key len = 0.105292 GB avg = 0.025 KB total ukey len = 0.071392 GB avg = 0.017 KB total ukey num = 0.004237542 Billion total entry num = 0.004237542 Billion write speed all = 19.592509457 MB/sec (with seq num) write speed all = 19.226670799 MB/sec (without seq num) 2024/11/19-15:16:27.399189 139851169007360 [compaction_job.cc:1948] [default] [JOB 41] Generated table #4438: 463513 keys, 55508593 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:27.399240 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000587399217, "cf_name": "default", "job": 41, "event": "table_file_creation", "file_number": 4438, "file_size": 55508593, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54626320, "index_size": 876800, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12978364, "raw_average_key_size": 28, "raw_value_size": 54130642, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463513, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000587, "oldest_key_time": 0, "file_creation_time": 1732000587, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4438, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:27.414035 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004440.sst) /storage/mytopling/..//tmp/Topling-82038-C0gu9I 2024/11/19-15:16:27.421222 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004439.sst): first pass time = 0.26's, 261.050'MB/sec 2024/11/19-15:16:27.421265 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004439.sst): old prealloc_size = 73819750, real_size = 55470912 2024/11/19-15:16:27.421509 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004439.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462168 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.80 avg-zval = 117.86 2024/11/19-15:16:27.421565 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004439.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.104 sec, Key+Value = 0.067 GB 2024/11/19-15:16:27.511204 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004439.sst): second pass time = 0.00's,166607419.753'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 615.666'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3633.815'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 2.525'MB/sec write SST data time = 0.04's, 1260.467'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 166607419.753'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462168 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.80 avg-zval = 117.86 usrkeys = 462168 avg-key = 20.00 avg-zkey = 1.92 avg-val = 116.80 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6597 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0554 GB } UnZip/Zip{ index = 10.4378 value = 0.9910 dict = 0.00 all = 1.2088 } Zip/UnZip{ index = 0.0958 value = 1.0091 dict = 0.00 all = 0.8272 } ---------------------------- total value len = 1.764223 GB avg = 0.375 KB (by entry num) total key len = 0.118233 GB avg = 0.025 KB total ukey len = 0.080635 GB avg = 0.017 KB total ukey num = 0.004699710 Billion total entry num = 0.004699710 Billion write speed all = 20.289336066 MB/sec (with seq num) write speed all = 19.884103743 MB/sec (without seq num) 2024/11/19-15:16:27.515112 139851179493120 [compaction_job.cc:1948] [default] [JOB 40] Generated table #4439: 462168 keys, 55360625 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:27.515164 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000587515141, "cf_name": "default", "job": 40, "event": "table_file_creation", "file_number": 4439, "file_size": 55360625, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54469616, "index_size": 885568, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12940704, "raw_average_key_size": 28, "raw_value_size": 53980804, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462168, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000587, "oldest_key_time": 0, "file_creation_time": 1732000587, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4439, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:27.529709 139851179493120 [compaction_job.cc:703] job-00040: subcompact[0], size: 0.110855 G, files: /storage/mytopling/.rocksdb [004437,004439] 2024/11/19-15:16:27.531029 139851179493120 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:16:27.531183 139851179493120 (Original Log Time 2024/11/19-15:16:26.795708) EVENT_LOG_v1 {"time_micros": 1732000586795689, "job": 40, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4356, 4363, 4370, 4373], "score": 3.88095, "input_data_size": 133300300, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:27.531191 139851179493120 (Original Log Time 2024/11/19-15:16:27.530647) [compaction_job.cc:2022] [default] [JOB 40] Compacted 4@1 files to L2 => 110855458 bytes 2024/11/19-15:16:27.531194 139851179493120 (Original Log Time 2024/11/19-15:16:27.530996) [compaction_job.cc:1224] [default] compacted to: files[5 59 67 75 0 0 0] max score 2.35 (107 files need compaction), MB/sec: 181.6 rd, 151.0 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925517, records dropped: 0 output_compression: Snappy 2024/11/19-15:16:27.531213 139851179493120 (Original Log Time 2024/11/19-15:16:27.531021) EVENT_LOG_v1 {"time_micros": 1732000587531010, "cf": "default", "job": 40, "event": "compaction_finished", "compaction_time_micros": 734015, "compaction_time_cpu_micros": 731024, "output_level": 2, "num_output_files": 2, "total_output_size": 110855458, "num_input_records": 925517, "num_output_records": 925517, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 59, 67, 75, 0, 0, 0]} 2024/11/19-15:16:27.533822 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004440.sst: sumWaitingMem = 0.005 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0049 GB, waited 0.000 sec, Key+Value = 0.055 GB 2024/11/19-15:16:27.535028 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004364.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:27.535053 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000587535050, "job": 40, "event": "table_file_deletion", "file_number": 4364} 2024/11/19-15:16:27.535195 139851179493120 [compaction_job.cc:2437] [default] [JOB 49] Compacting 1@2 files to L3, score 2.35, subcompactions 1 : 1 2024/11/19-15:16:27.535203 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 47 Base level 2, inputs: [4424(2218B)] 2024/11/19-15:16:27.574307 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004440.sst): index pass time = 0.04's, 150.522'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 380588 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0061 GB zip-key = 0.0007 GB avg-key = 16.00 avg-zkey = 1.91 2024/11/19-15:16:27.628113 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004440.sst): first pass time = 0.21's, 257.399'MB/sec 2024/11/19-15:16:27.628171 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004440.sst): old prealloc_size = 73819750, real_size = 45673072 2024/11/19-15:16:27.628375 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004440.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 380588 prefix = 4 raw-key = 0.0061 GB zip-key = 0.0007 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0444 GB zip-val = 0.0449 GB avg-val = 116.77 avg-zval = 117.85 2024/11/19-15:16:27.628445 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004440.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0009 GB, waited 0.095 sec, Key+Value = 0.055 GB 2024/11/19-15:16:27.686453 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-00 2024/11/19-15:16:27.686527 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00045/att-00/compact.done) = No such file or directory: 5.177 sec on http probe failed: 2, retry 2024/11/19-15:16:27.701652 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004440.sst): second pass time = 0.00's,109463605.911'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 620.421'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3654.674'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.547'MB/sec write SST data time = 0.04's, 1269.308'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.044'GB zip my value throughput = 109463605.911'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 380588 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.77 avg-zval = 117.85 usrkeys = 380588 avg-key = 20.00 avg-zkey = 1.91 avg-val = 116.77 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 5433 UnZipSize{ index = 0.0076 GB value = 0.0444 GB dict = 0.00 MB all = 0.0551 GB } __ZipSize{ index = 0.0007 GB value = 0.0449 GB dict = 0.00 MB all = 0.0456 GB } UnZip/Zip{ index = 10.4935 value = 0.9909 dict = 0.00 all = 1.2088 } Zip/UnZip{ index = 0.0953 value = 1.0092 dict = 0.00 all = 0.8273 } ---------------------------- total value len = 1.808665 GB avg = 0.356 KB (by entry num) total key len = 0.128889 GB avg = 0.025 KB total ukey len = 0.088247 GB avg = 0.017 KB total ukey num = 0.005080298 Billion total entry num = 0.005080298 Billion write speed all = 20.840416960 MB/sec (with seq num) write speed all = 20.403265796 MB/sec (without seq num) 2024/11/19-15:16:27.704729 139851169007360 [compaction_job.cc:1948] [default] [JOB 41] Generated table #4440: 380588 keys, 45582449 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:27.704779 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000587704756, "cf_name": "default", "job": 41, "event": "table_file_creation", "file_number": 4440, "file_size": 45582449, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 44852384, "index_size": 725376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 10656464, "raw_average_key_size": 28, "raw_value_size": 44442224, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 380588, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000587, "oldest_key_time": 0, "file_creation_time": 1732000587, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4440, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:27.718367 139851169007360 [compaction_job.cc:703] job-00041: subcompact[0], size: 0.101091 G, files: /storage/mytopling/.rocksdb [004438,004440] 2024/11/19-15:16:27.719657 139851169007360 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:16:27.720660 139851169007360 (Original Log Time 2024/11/19-15:16:27.039340) EVENT_LOG_v1 {"time_micros": 1732000587039318, "job": 41, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4378, 4382, 4387, 4391], "score": 3.6313, "input_data_size": 121592204, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:27.720671 139851169007360 (Original Log Time 2024/11/19-15:16:27.719283) [compaction_job.cc:2022] [default] [JOB 41] Compacted 4@1 files to L2 => 101091042 bytes 2024/11/19-15:16:27.720675 139851169007360 (Original Log Time 2024/11/19-15:16:27.719623) [compaction_job.cc:1224] [default] compacted to: files[5 55 69 75 0 0 0] max score 2.42 (108 files need compaction), MB/sec: 179.1 rd, 148.9 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(116.0, 0.0 +0.0 blob) out(96.4 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 844101, records dropped: 0 output_compression: Snappy 2024/11/19-15:16:27.720701 139851169007360 (Original Log Time 2024/11/19-15:16:27.719649) EVENT_LOG_v1 {"time_micros": 1732000587719636, "cf": "default", "job": 41, "event": "compaction_finished", "compaction_time_micros": 679043, "compaction_time_cpu_micros": 675048, "output_level": 2, "num_output_files": 2, "total_output_size": 101091042, "num_input_records": 844101, "num_output_records": 844101, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 55, 69, 75, 0, 0, 0]} 2024/11/19-15:16:27.724766 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003809.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:27.724792 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000587724788, "job": 41, "event": "table_file_deletion", "file_number": 3809} 2024/11/19-15:16:27.724968 139851169007360 [compaction_job.cc:2437] [default] [JOB 50] Compacting 1@2 files to L3, score 2.42, subcompactions 1 : 1 2024/11/19-15:16:27.724977 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 48 Base level 2, inputs: [4426(17KB)] 2024/11/19-15:16:27.798790 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-00 2024/11/19-15:16:27.798879 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00046/att-00/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:16:27.817957 139851179493120 [dcompact_etcd.cc:1163] job-00049/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.282 sec, input = 650 B 2024/11/19-15:16:27.827498 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-00 2024/11/19-15:16:27.827562 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00049/att-00/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:16:27.832658 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-00 2024/11/19-15:16:27.832706 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00049/att-00/compact.done) = No such file or directory: 0.010 sec on http probe failed: 2, retry 2024/11/19-15:16:27.840460 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-00 2024/11/19-15:16:27.840506 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00049/att-00/compact.done) = No such file or directory: 0.015 sec on http probe failed: 3, retry 2024/11/19-15:16:27.845311 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-00 2024/11/19-15:16:27.845357 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00049/att-00/compact.done) = No such file or directory: 0.023 sec on http probe failed: 4, retry 2024/11/19-15:16:27.852587 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-00 2024/11/19-15:16:27.852633 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00049/att-00/compact.done) = No such file or directory: 0.027 sec on http probe failed: 5, giveup 2024/11/19-15:16:27.852647 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00049/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.317 (submit 282.280 ms wait 0.027 s), input 650 B, send shutdown 2024/11/19-15:16:27.862588 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-00 2024/11/19-15:16:27.862820 139851179493120 [dcompact_etcd.cc:1773] job-00049/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:28.015331 139851169007360 [dcompact_etcd.cc:1163] job-00050/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.290 sec, input = 8.148 KiB 2024/11/19-15:16:28.020784 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-00 2024/11/19-15:16:28.020847 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00050/att-00/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:16:28.028725 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-00 2024/11/19-15:16:28.028772 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00050/att-00/compact.done) = No such file or directory: 0.006 sec on http probe failed: 2, retry 2024/11/19-15:16:28.038052 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-00 2024/11/19-15:16:28.038099 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00050/att-00/compact.done) = No such file or directory: 0.014 sec on http probe failed: 3, retry 2024/11/19-15:16:28.044131 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-00 2024/11/19-15:16:28.044175 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00050/att-00/compact.done) = No such file or directory: 0.023 sec on http probe failed: 4, retry 2024/11/19-15:16:28.055189 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-00 2024/11/19-15:16:28.055298 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00050/att-00/compact.done) = No such file or directory: 0.029 sec on http probe failed: 5, giveup 2024/11/19-15:16:28.055314 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00050/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.330 (submit 289.855 ms wait 0.029 s), input 8.148 KiB, send shutdown 2024/11/19-15:16:28.059781 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-00 2024/11/19-15:16:28.060042 139851169007360 [dcompact_etcd.cc:1773] job-00050/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:28.143989 139851179493120 [dcompact_etcd.cc:1163] job-00049/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 650 B 2024/11/19-15:16:28.153076 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-01 2024/11/19-15:16:28.153138 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00049/att-01/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:16:28.160941 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-01 2024/11/19-15:16:28.160985 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00049/att-01/compact.done) = No such file or directory: 0.009 sec on http probe failed: 2, retry 2024/11/19-15:16:28.165363 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-01 2024/11/19-15:16:28.165404 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00049/att-01/compact.done) = No such file or directory: 0.017 sec on http probe failed: 3, retry 2024/11/19-15:16:28.174298 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-01 2024/11/19-15:16:28.174339 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00049/att-01/compact.done) = No such file or directory: 0.022 sec on http probe failed: 4, retry 2024/11/19-15:16:28.182828 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-01 2024/11/19-15:16:28.182869 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00049/att-01/compact.done) = No such file or directory: 0.030 sec on http probe failed: 5, giveup 2024/11/19-15:16:28.182882 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00049/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.320 (submit 281.073 ms wait 0.030 s), input 650 B, send shutdown 2024/11/19-15:16:28.190312 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-01 2024/11/19-15:16:28.190513 139851179493120 [dcompact_etcd.cc:1773] job-00049/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:28.241088 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-00 2024/11/19-15:16:28.241145 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00047/att-00/compact.done) = No such file or directory: 5.056 sec on http probe failed: 2, retry 2024/11/19-15:16:28.351857 139851169007360 [dcompact_etcd.cc:1163] job-00050/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.292 sec, input = 8.148 KiB 2024/11/19-15:16:28.362030 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-01 2024/11/19-15:16:28.362092 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00050/att-01/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:16:28.369593 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-01 2024/11/19-15:16:28.369638 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00050/att-01/compact.done) = No such file or directory: 0.010 sec on http probe failed: 2, retry 2024/11/19-15:16:28.378035 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-01 2024/11/19-15:16:28.378079 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00050/att-01/compact.done) = No such file or directory: 0.018 sec on http probe failed: 3, retry 2024/11/19-15:16:28.387652 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-01 2024/11/19-15:16:28.387695 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00050/att-01/compact.done) = No such file or directory: 0.026 sec on http probe failed: 4, retry 2024/11/19-15:16:28.393243 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-01 2024/11/19-15:16:28.393285 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00050/att-01/compact.done) = No such file or directory: 0.036 sec on http probe failed: 5, giveup 2024/11/19-15:16:28.393298 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00050/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.333 (submit 291.714 ms wait 0.036 s), input 8.148 KiB, send shutdown 2024/11/19-15:16:28.402466 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-01 2024/11/19-15:16:28.402680 139851169007360 [dcompact_etcd.cc:1773] job-00050/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:28.468876 139851179493120 [dcompact_etcd.cc:1163] job-00049/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.278 sec, input = 650 B 2024/11/19-15:16:28.477459 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-02 2024/11/19-15:16:28.477516 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00049/att-02/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:16:28.482363 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-02 2024/11/19-15:16:28.482409 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00049/att-02/compact.done) = No such file or directory: 0.009 sec on http probe failed: 2, retry 2024/11/19-15:16:28.487385 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-02 2024/11/19-15:16:28.487431 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00049/att-02/compact.done) = No such file or directory: 0.014 sec on http probe failed: 3, retry 2024/11/19-15:16:28.503817 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-02 2024/11/19-15:16:28.503863 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00049/att-02/compact.done) = No such file or directory: 0.019 sec on http probe failed: 4, retry 2024/11/19-15:16:28.512966 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-00 2024/11/19-15:16:28.513019 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00044/att-00/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:16:28.513191 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-02 2024/11/19-15:16:28.513241 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00049/att-02/compact.done) = No such file or directory: 0.035 sec on http probe failed: 5, giveup 2024/11/19-15:16:28.513254 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00049/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.323 (submit 278.273 ms wait 0.035 s), input 650 B, send shutdown 2024/11/19-15:16:28.519084 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00049/att-02 2024/11/19-15:16:28.519327 139851179493120 [dcompact_etcd.cc:1773] job-00049/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:28.521084 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00049) = success 2024/11/19-15:16:28.521092 139851179493120 [dcompact_etcd.cc:1773] job-00049/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:16:28.521109 139851179493120 (Original Log Time 2024/11/19-15:16:27.535215) EVENT_LOG_v1 {"time_micros": 1732000587535208, "job": 49, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4424], "score": 2.34713, "input_data_size": 2218, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:28.521118 139851179493120 [compaction_job.cc:2437] [default] [JOB 49] Compacting 1@2 files to L3, score 2.35, subcompactions 1 : 1 2024/11/19-15:16:28.521124 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 47 Base level 2, inputs: [4424(2218B)] 2024/11/19-15:16:28.521271 139851179493120 [compaction_job.cc:703] job-00049: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:16:28.521647 139851179493120 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:16:28.521742 139851179493120 (Original Log Time 2024/11/19-15:16:28.521151) EVENT_LOG_v1 {"time_micros": 1732000588521136, "job": 49, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4424], "score": 2.34713, "input_data_size": 2218, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:28.521747 139851179493120 (Original Log Time 2024/11/19-15:16:28.521322) [compaction_job.cc:2022] [default] [JOB 49] Compacted 1@2 files to L3 => 0 bytes 2024/11/19-15:16:28.521750 139851179493120 (Original Log Time 2024/11/19-15:16:28.521623) [compaction_job.cc:1224] [default] compacted to: files[5 55 68 75 0 0 0] max score 2.42 (107 files need compaction), MB/sec: 17.7 rd, 0.0 wr, level 3, files in(1, 0) out(0 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 5, records dropped: 5 output_compression: Snappy 2024/11/19-15:16:28.521766 139851179493120 (Original Log Time 2024/11/19-15:16:28.521641) EVENT_LOG_v1 {"time_micros": 1732000588521632, "cf": "default", "job": 49, "event": "compaction_finished", "compaction_time_micros": 125, "compaction_time_cpu_micros": 112, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 5, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 55, 68, 75, 0, 0, 0]} 2024/11/19-15:16:28.521841 139851179493120 [compaction_job.cc:2437] [default] [JOB 51] Compacting 2@2 files to L3, score 2.42, subcompactions 1 : 1 2024/11/19-15:16:28.521848 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 49 Base level 2, inputs: [4433(62MB) 4434(49MB)] 2024/11/19-15:16:28.702396 139851169007360 [dcompact_etcd.cc:1163] job-00050/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.300 sec, input = 8.148 KiB 2024/11/19-15:16:28.712096 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-02 2024/11/19-15:16:28.712164 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00050/att-02/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:16:28.719901 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-02 2024/11/19-15:16:28.719956 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00050/att-02/compact.done) = No such file or directory: 0.010 sec on http probe failed: 2, retry 2024/11/19-15:16:28.728211 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-02 2024/11/19-15:16:28.728270 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00050/att-02/compact.done) = No such file or directory: 0.018 sec on http probe failed: 3, retry 2024/11/19-15:16:28.735138 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-02 2024/11/19-15:16:28.735204 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00042/att-02/compact.done) = No such file or directory: 9.200 sec on http probe failed: 4, retry 2024/11/19-15:16:28.737627 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-02 2024/11/19-15:16:28.737678 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00050/att-02/compact.done) = No such file or directory: 0.026 sec on http probe failed: 4, retry 2024/11/19-15:16:28.742582 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-02 2024/11/19-15:16:28.742627 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00050/att-02/compact.done) = No such file or directory: 0.036 sec on http probe failed: 5, giveup 2024/11/19-15:16:28.742641 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00050/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.340 (submit 299.615 ms wait 0.036 s), input 8.148 KiB, send shutdown 2024/11/19-15:16:28.750071 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00050/att-02 2024/11/19-15:16:28.750305 139851169007360 [dcompact_etcd.cc:1773] job-00050/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:28.752061 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00050) = success 2024/11/19-15:16:28.752070 139851169007360 [dcompact_etcd.cc:1773] job-00050/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:16:28.752088 139851169007360 (Original Log Time 2024/11/19-15:16:27.724990) EVENT_LOG_v1 {"time_micros": 1732000587724982, "job": 50, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4426], "score": 2.423, "input_data_size": 18282, "oldest_snapshot_seqno": 112663567} 2024/11/19-15:16:28.752095 139851169007360 [compaction_job.cc:2437] [default] [JOB 50] Compacting 1@2 files to L3, score 2.42, subcompactions 1 : 1 2024/11/19-15:16:28.752103 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 48 Base level 2, inputs: [4426(17KB)] 2024/11/19-15:16:28.752268 139851169007360 [compaction_job.cc:703] job-00050: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:16:28.752649 139851169007360 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:16:28.752756 139851169007360 (Original Log Time 2024/11/19-15:16:28.752134) EVENT_LOG_v1 {"time_micros": 1732000588752116, "job": 50, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4426], "score": 2.423, "input_data_size": 18282, "oldest_snapshot_seqno": 112663567} 2024/11/19-15:16:28.752764 139851169007360 (Original Log Time 2024/11/19-15:16:28.752313) [compaction_job.cc:2022] [default] [JOB 50] Compacted 1@2 files to L3 => 0 bytes 2024/11/19-15:16:28.752769 139851169007360 (Original Log Time 2024/11/19-15:16:28.752627) [compaction_job.cc:1224] [default] compacted to: files[5 55 67 75 0 0 0] max score 2.35 (105 files need compaction), MB/sec: 131.5 rd, 0.0 wr, level 3, files in(1, 0) out(0 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 60, records dropped: 60 output_compression: Snappy 2024/11/19-15:16:28.752789 139851169007360 (Original Log Time 2024/11/19-15:16:28.752643) EVENT_LOG_v1 {"time_micros": 1732000588752634, "cf": "default", "job": 50, "event": "compaction_finished", "compaction_time_micros": 139, "compaction_time_cpu_micros": 126, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 60, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 55, 67, 75, 0, 0, 0]} 2024/11/19-15:16:28.752887 139851169007360 [compaction_job.cc:2437] [default] [JOB 52] Compacting 1@2 files to L3, score 2.35, subcompactions 1 : 1 2024/11/19-15:16:28.752894 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 50 Base level 2, inputs: [4430(10017KB)] 2024/11/19-15:16:28.814603 139851179493120 [dcompact_etcd.cc:1163] job-00051/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.292 sec, input = 114.819 MiB 2024/11/19-15:16:29.047950 139851169007360 [dcompact_etcd.cc:1163] job-00052/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.295 sec, input = 12.700 MiB 2024/11/19-15:16:29.374067 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-00 2024/11/19-15:16:29.374149 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00052/att-00/compact.done) = No such file or directory: 0.318 sec on http probe failed: 1, retry 2024/11/19-15:16:29.636711 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-00 2024/11/19-15:16:29.636789 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00052/att-00/compact.done) = No such file or directory: 0.580 sec on http probe failed: 2, retry 2024/11/19-15:16:29.695576 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-00 2024/11/19-15:16:29.695671 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00045/att-00/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:16:29.806337 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-00 2024/11/19-15:16:29.806434 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00046/att-00/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:16:29.898006 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-00 2024/11/19-15:16:29.898099 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00052/att-00/compact.done) = No such file or directory: 0.843 sec on http probe failed: 3, retry 2024/11/19-15:16:30.157626 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-00 2024/11/19-15:16:30.157708 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00052/att-00/compact.done) = No such file or directory: 1.104 sec on http probe failed: 4, retry 2024/11/19-15:16:30.251003 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-00 2024/11/19-15:16:30.251086 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00047/att-00/compact.done) = No such file or directory: 7.063 sec on http probe failed: 3, retry 2024/11/19-15:16:30.421329 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-00 2024/11/19-15:16:30.421434 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00052/att-00/compact.done) = No such file or directory: 1.364 sec on http probe failed: 5, giveup 2024/11/19-15:16:30.421460 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00052/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 12.700 sec, real 1.668 (submit 294.585 ms wait 1.364 s), input 12.700 MiB, send shutdown 2024/11/19-15:16:30.431094 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-00 2024/11/19-15:16:30.431586 139851169007360 [dcompact_etcd.cc:1773] job-00052/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:30.522749 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-00 2024/11/19-15:16:30.522824 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00044/att-00/compact.done) = No such file or directory: 9.204 sec on http probe failed: 4, retry 2024/11/19-15:16:30.728505 139851169007360 [dcompact_etcd.cc:1163] job-00052/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.297 sec, input = 12.700 MiB 2024/11/19-15:16:30.739795 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-02 2024/11/19-15:16:30.739856 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00042/att-02/compact.done) = No such file or directory: 11.205 sec on http probe failed: 5, giveup 2024/11/19-15:16:30.739873 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00042/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.879 sec, real 11.510 (submit 300.913 ms wait 11.205 s), input 126.879 MiB, send shutdown 2024/11/19-15:16:30.748080 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00042/att-02 2024/11/19-15:16:30.748297 139851127064320 [dcompact_etcd.cc:1773] job-00042/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:30.751409 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00042) = success 2024/11/19-15:16:30.751417 139851127064320 [dcompact_etcd.cc:1773] job-00042/att-03: CleanFiles 0.003 sec : 0.000 2024/11/19-15:16:30.751437 139851127064320 (Original Log Time 2024/11/19-15:15:56.165315) EVENT_LOG_v1 {"time_micros": 1732000556165310, "job": 42, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4394, 4396, 4398, 4400], "score": 3.40367, "input_data_size": 134100352, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:30.751444 139851127064320 [compaction_job.cc:2437] [default] [JOB 42] Compacting 4@1 files to L2, score 3.40, subcompactions 1 : 1 2024/11/19-15:16:30.751452 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 40 Base level 1, inputs: [4394(31MB) 4396(31MB) 4398(31MB) 4400(31MB)] 2024/11/19-15:16:30.751680 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004441.sst) /storage/mytopling/..//tmp/Topling-82038-olEwcm 2024/11/19-15:16:30.805548 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004441.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:16:30.808862 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004441.sst): index pass time = 0.00's, 157.617'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64717 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:16:30.871902 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004441.sst): first pass time = 0.12's, 558.275'MB/sec 2024/11/19-15:16:30.871962 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004441.sst): old prealloc_size = 73819750, real_size = 66152272 2024/11/19-15:16:30.872019 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004441.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64717 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.97 avg-zval =1020.22 2024/11/19-15:16:30.872071 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004441.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.067 GB 2024/11/19-15:16:30.931702 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004441.sst): second pass time = 0.00's,148903104.072'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1107.400'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2117.267'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.376'MB/sec write SST data time = 0.02's, 3979.076'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 148903104.072'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64717 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.97 avg-zval = 1020.22 usrkeys = 64717 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.97 avg-zval = 1020.22 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8063 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0467 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 1.874481 GB avg = 0.364 KB (by entry num) total key len = 0.130184 GB avg = 0.025 KB total ukey len = 0.089023 GB avg = 0.017 KB total ukey num = 0.005145015 Billion total entry num = 0.005145015 Billion write speed all = 20.838275504 MB/sec (with seq num) write speed all = 20.410420317 MB/sec (without seq num) 2024/11/19-15:16:30.935410 139851127064320 [compaction_job.cc:1948] [default] [JOB 42] Generated table #4441: 64717 keys, 66137453 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:30.935464 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000590935440, "cf_name": "default", "job": 42, "event": "table_file_creation", "file_number": 4441, "file_size": 66137453, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025728, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294340, "raw_average_key_size": 20, "raw_value_size": 65815172, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64717, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000590, "oldest_key_time": 0, "file_creation_time": 1732000590, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4441, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:30.950172 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004442.sst) /storage/mytopling/..//tmp/Topling-82038-97PUzo 2024/11/19-15:16:31.001780 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004442.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:16:31.001881 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004442.sst): index pass time = 0.00's,6973.130'MB/sec index type = UintIndex_AllOne usrkeys = 63581 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:16:31.054798 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-01 2024/11/19-15:16:31.054902 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00052/att-01/compact.done) = No such file or directory: 0.318 sec on http probe failed: 1, retry 2024/11/19-15:16:31.068084 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004442.sst): first pass time = 0.12's, 559.251'MB/sec 2024/11/19-15:16:31.068129 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004442.sst): old prealloc_size = 73819750, real_size = 64883880 2024/11/19-15:16:31.068150 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004442.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63581 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1016.98 avg-zval =1020.24 2024/11/19-15:16:31.120896 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004442.sst): second pass time = 0.00's,153588534.442'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1229.820'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.209'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.416'MB/sec write SST data time = 0.01's, 4480.989'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 153588534.442'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63581 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.98 avg-zval = 1020.24 usrkeys = 63581 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.98 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0000 GB value = 0.0649 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index =3973.8125 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 1.939141 GB avg = 0.372 KB (by entry num) total key len = 0.131455 GB avg = 0.025 KB total ukey len = 0.089786 GB avg = 0.017 KB total ukey num = 0.005208596 Billion total entry num = 0.005208596 Billion write speed all = 21.481390040 MB/sec (with seq num) write speed all = 21.049097667 MB/sec (without seq num) 2024/11/19-15:16:31.124825 139851127064320 [compaction_job.cc:1948] [default] [JOB 42] Generated table #4442: 63581 keys, 64868915 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:31.124877 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000591124853, "cf_name": "default", "job": 42, "event": "table_file_creation", "file_number": 4442, "file_size": 64868915, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64867632, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1271620, "raw_average_key_size": 20, "raw_value_size": 64660773, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63581, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000590, "oldest_key_time": 0, "file_creation_time": 1732000590, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4442, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:31.139071 139851127064320 [compaction_job.cc:703] job-00042: subcompact[0], size: 0.131006 G, files: /storage/mytopling/.rocksdb [004441,004442] 2024/11/19-15:16:31.139839 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:16:31.140018 139851127064320 (Original Log Time 2024/11/19-15:16:30.751482) EVENT_LOG_v1 {"time_micros": 1732000590751464, "job": 42, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4394, 4396, 4398, 4400], "score": 3.40367, "input_data_size": 134100352, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:31.140025 139851127064320 (Original Log Time 2024/11/19-15:16:31.139486) [compaction_job.cc:2022] [default] [JOB 42] Compacted 4@1 files to L2 => 131006368 bytes 2024/11/19-15:16:31.140028 139851127064320 (Original Log Time 2024/11/19-15:16:31.139810) [compaction_job.cc:1224] [default] compacted to: files[5 51 69 75 0 0 0] max score 2.42 (106 files need compaction), MB/sec: 346.0 rd, 338.0 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(124.9 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128298, records dropped: 0 output_compression: Snappy 2024/11/19-15:16:31.140047 139851127064320 (Original Log Time 2024/11/19-15:16:31.139833) EVENT_LOG_v1 {"time_micros": 1732000591139821, "cf": "default", "job": 42, "event": "compaction_finished", "compaction_time_micros": 387601, "compaction_time_cpu_micros": 386034, "output_level": 2, "num_output_files": 2, "total_output_size": 131006368, "num_input_records": 128298, "num_output_records": 128298, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 51, 69, 75, 0, 0, 0]} 2024/11/19-15:16:31.144110 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004355.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:31.144136 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000591144134, "job": 42, "event": "table_file_deletion", "file_number": 4355} 2024/11/19-15:16:31.148128 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004358.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:31.148150 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000591148148, "job": 42, "event": "table_file_deletion", "file_number": 4358} 2024/11/19-15:16:31.152167 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004360.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:31.152187 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000591152185, "job": 42, "event": "table_file_deletion", "file_number": 4360} 2024/11/19-15:16:31.154766 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004362.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:31.154785 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000591154783, "job": 42, "event": "table_file_deletion", "file_number": 4362} 2024/11/19-15:16:31.154940 139851127064320 [compaction_job.cc:2437] [default] [JOB 53] Compacting 1@2 files to L3, score 2.42, subcompactions 1 : 1 2024/11/19-15:16:31.154948 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 51 Base level 2, inputs: [4429(3939KB)] 2024/11/19-15:16:31.316805 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-01 2024/11/19-15:16:31.316899 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00052/att-01/compact.done) = No such file or directory: 0.580 sec on http probe failed: 2, retry 2024/11/19-15:16:31.482286 139851127064320 [dcompact_etcd.cc:1163] job-00053/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.327 sec, input = 7.519 MiB 2024/11/19-15:16:31.579457 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-01 2024/11/19-15:16:31.579544 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00052/att-01/compact.done) = No such file or directory: 0.842 sec on http probe failed: 3, retry 2024/11/19-15:16:31.679365 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-00 2024/11/19-15:16:31.679465 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00053/att-00/compact.done) = No such file or directory: 0.188 sec on http probe failed: 1, retry 2024/11/19-15:16:31.693257 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-00 2024/11/19-15:16:31.693325 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00051/att-00/compact.done) = No such file or directory: 2.871 sec on http probe failed: 1, retry 2024/11/19-15:16:31.699868 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-00 2024/11/19-15:16:31.699925 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00045/att-00/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:16:31.812216 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-00 2024/11/19-15:16:31.812305 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00046/att-00/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:16:31.837737 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-00 2024/11/19-15:16:31.837796 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00053/att-00/compact.done) = No such file or directory: 0.348 sec on http probe failed: 2, retry 2024/11/19-15:16:31.839036 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-01 2024/11/19-15:16:31.839091 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00052/att-01/compact.done) = No such file or directory: 1.105 sec on http probe failed: 4, retry 2024/11/19-15:16:31.996401 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-00 2024/11/19-15:16:31.996470 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00053/att-00/compact.done) = No such file or directory: 0.506 sec on http probe failed: 3, retry 2024/11/19-15:16:32.099504 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-01 2024/11/19-15:16:32.099571 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00052/att-01/compact.done) = No such file or directory: 1.365 sec on http probe failed: 5, giveup 2024/11/19-15:16:32.099588 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00052/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 12.700 sec, real 1.668 (submit 296.777 ms wait 1.365 s), input 12.700 MiB, send shutdown 2024/11/19-15:16:32.107648 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-01 2024/11/19-15:16:32.107869 139851169007360 [dcompact_etcd.cc:1773] job-00052/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:32.155260 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-00 2024/11/19-15:16:32.155315 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00053/att-00/compact.done) = No such file or directory: 0.665 sec on http probe failed: 4, retry 2024/11/19-15:16:32.257060 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-00 2024/11/19-15:16:32.257123 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00047/att-00/compact.done) = No such file or directory: 9.073 sec on http probe failed: 4, retry 2024/11/19-15:16:32.314914 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-00 2024/11/19-15:16:32.314980 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00053/att-00/compact.done) = No such file or directory: 0.823 sec on http probe failed: 5, giveup 2024/11/19-15:16:32.314995 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00053/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 7.519 sec, real 1.160 (submit 326.834 ms wait 0.823 s), input 7.519 MiB, send shutdown 2024/11/19-15:16:32.323126 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-00 2024/11/19-15:16:32.323347 139851127064320 [dcompact_etcd.cc:1773] job-00053/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:32.529214 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-00 2024/11/19-15:16:32.529286 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00044/att-00/compact.done) = No such file or directory: 11.213 sec on http probe failed: 5, giveup 2024/11/19-15:16:32.529304 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00044/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.907 sec, real 11.522 (submit 301.647 ms wait 11.213 s), input 126.907 MiB, send shutdown 2024/11/19-15:16:32.535422 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-00 2024/11/19-15:16:32.535640 139850978166528 [dcompact_etcd.cc:1773] job-00044/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:32.590266 139851169007360 [dcompact_etcd.cc:1163] job-00052/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.482 sec, input = 12.700 MiB 2024/11/19-15:16:32.603877 139851127064320 [dcompact_etcd.cc:1163] job-00053/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 7.519 MiB 2024/11/19-15:16:32.798418 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-01 2024/11/19-15:16:32.798522 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00053/att-01/compact.done) = No such file or directory: 0.188 sec on http probe failed: 1, retry 2024/11/19-15:16:32.820030 139850978166528 [dcompact_etcd.cc:1163] job-00044/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.284 sec, input = 126.907 MiB 2024/11/19-15:16:32.913546 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-02 2024/11/19-15:16:32.913631 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00052/att-02/compact.done) = No such file or directory: 0.318 sec on http probe failed: 1, retry 2024/11/19-15:16:32.957304 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-01 2024/11/19-15:16:32.957374 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00053/att-01/compact.done) = No such file or directory: 0.345 sec on http probe failed: 2, retry 2024/11/19-15:16:33.112641 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-01 2024/11/19-15:16:33.112707 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00053/att-01/compact.done) = No such file or directory: 0.504 sec on http probe failed: 3, retry 2024/11/19-15:16:33.176324 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-02 2024/11/19-15:16:33.176395 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00052/att-02/compact.done) = No such file or directory: 0.577 sec on http probe failed: 2, retry 2024/11/19-15:16:33.268710 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-01 2024/11/19-15:16:33.268769 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00053/att-01/compact.done) = No such file or directory: 0.659 sec on http probe failed: 4, retry 2024/11/19-15:16:33.425044 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-01 2024/11/19-15:16:33.425098 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00053/att-01/compact.done) = No such file or directory: 0.815 sec on http probe failed: 5, giveup 2024/11/19-15:16:33.425114 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00053/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 7.519 sec, real 1.102 (submit 280.431 ms wait 0.815 s), input 7.519 MiB, send shutdown 2024/11/19-15:16:33.433158 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-01 2024/11/19-15:16:33.433372 139851127064320 [dcompact_etcd.cc:1773] job-00053/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:33.435413 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-02 2024/11/19-15:16:33.435469 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00052/att-02/compact.done) = No such file or directory: 0.840 sec on http probe failed: 3, retry 2024/11/19-15:16:33.700163 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-02 2024/11/19-15:16:33.700226 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00052/att-02/compact.done) = No such file or directory: 1.099 sec on http probe failed: 4, retry 2024/11/19-15:16:33.701495 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-00 2024/11/19-15:16:33.701553 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00051/att-00/compact.done) = No such file or directory: 4.879 sec on http probe failed: 2, retry 2024/11/19-15:16:33.708421 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-00 2024/11/19-15:16:33.708475 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00045/att-00/compact.done) = No such file or directory: 11.199 sec on http probe failed: 5, giveup 2024/11/19-15:16:33.708491 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00045/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.825 sec, real 11.519 (submit 311.448 ms wait 11.199 s), input 126.825 MiB, send shutdown 2024/11/19-15:16:33.721687 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-00 2024/11/19-15:16:33.721887 139851148035840 [dcompact_etcd.cc:1773] job-00045/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:33.728651 139851127064320 [dcompact_etcd.cc:1163] job-00053/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.295 sec, input = 7.519 MiB 2024/11/19-15:16:33.817014 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-00 2024/11/19-15:16:33.817069 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00046/att-00/compact.done) = No such file or directory: 11.201 sec on http probe failed: 5, giveup 2024/11/19-15:16:33.817085 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00046/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.836 sec, real 11.513 (submit 307.966 ms wait 11.201 s), input 126.836 MiB, send shutdown 2024/11/19-15:16:33.824932 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-00 2024/11/19-15:16:33.825129 139851158521600 [dcompact_etcd.cc:1773] job-00046/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:33.922805 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-02 2024/11/19-15:16:33.922905 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00053/att-02/compact.done) = No such file or directory: 0.188 sec on http probe failed: 1, retry 2024/11/19-15:16:33.964231 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-02 2024/11/19-15:16:33.964351 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00052/att-02/compact.done) = No such file or directory: 1.364 sec on http probe failed: 5, giveup 2024/11/19-15:16:33.964377 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00052/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 12.700 sec, real 1.856 (submit 482.290 ms wait 1.364 s), input 12.700 MiB, send shutdown 2024/11/19-15:16:33.974270 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00052/att-02 2024/11/19-15:16:33.974538 139851169007360 [dcompact_etcd.cc:1773] job-00052/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:33.976332 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00052) = success 2024/11/19-15:16:33.976345 139851169007360 [dcompact_etcd.cc:1773] job-00052/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:16:33.976371 139851169007360 (Original Log Time 2024/11/19-15:16:28.752905) EVENT_LOG_v1 {"time_micros": 1732000588752900, "job": 52, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4430], "score": 2.34824, "input_data_size": 10258164, "oldest_snapshot_seqno": 112716062} 2024/11/19-15:16:33.976381 139851169007360 [compaction_job.cc:2437] [default] [JOB 52] Compacting 1@2 files to L3, score 2.35, subcompactions 1 : 1 2024/11/19-15:16:33.976392 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 50 Base level 2, inputs: [4430(10017KB)] 2024/11/19-15:16:34.001394 139851169007360 [compaction_job.cc:703] job-00052: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:16:34.001801 139851169007360 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:16:34.001895 139851169007360 (Original Log Time 2024/11/19-15:16:33.976430) EVENT_LOG_v1 {"time_micros": 1732000593976408, "job": 52, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4430], "score": 2.34824, "input_data_size": 10258164, "oldest_snapshot_seqno": 112716062} 2024/11/19-15:16:34.001901 139851169007360 (Original Log Time 2024/11/19-15:16:34.001453) [compaction_job.cc:2022] [default] [JOB 52] Compacted 1@2 files to L3 => 0 bytes 2024/11/19-15:16:34.001904 139851169007360 (Original Log Time 2024/11/19-15:16:34.001769) [compaction_job.cc:1224] [default] compacted to: files[5 51 68 75 0 0 0] max score 2.42 (105 files need compaction), MB/sec: 410.7 rd, 0.0 wr, level 3, files in(1, 0) out(0 +0 blob) MB in(9.8, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 180000, records dropped: 180000 output_compression: Snappy 2024/11/19-15:16:34.001933 139851169007360 (Original Log Time 2024/11/19-15:16:34.001794) EVENT_LOG_v1 {"time_micros": 1732000594001781, "cf": "default", "job": 52, "event": "compaction_finished", "compaction_time_micros": 24977, "compaction_time_cpu_micros": 24857, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 180000, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 51, 68, 75, 0, 0, 0]} 2024/11/19-15:16:34.002021 139851169007360 [compaction_job.cc:2437] [default] [JOB 54] Compacting 4@2 files to L3, score 2.42, subcompactions 1 : 1 2024/11/19-15:16:34.002029 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 52 Base level 2, inputs: [4437(52MB) 4439(52MB) 4438(52MB) 4440(43MB)] 2024/11/19-15:16:34.015401 139851148035840 [dcompact_etcd.cc:1163] job-00045/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.293 sec, input = 126.825 MiB 2024/11/19-15:16:34.078270 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-02 2024/11/19-15:16:34.078332 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00053/att-02/compact.done) = No such file or directory: 0.345 sec on http probe failed: 2, retry 2024/11/19-15:16:34.094750 139851158521600 [dcompact_etcd.cc:1163] job-00046/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.270 sec, input = 126.836 MiB 2024/11/19-15:16:34.237392 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-02 2024/11/19-15:16:34.237455 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00053/att-02/compact.done) = No such file or directory: 0.500 sec on http probe failed: 3, retry 2024/11/19-15:16:34.269270 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-00 2024/11/19-15:16:34.269335 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00047/att-00/compact.done) = No such file or directory: 11.079 sec on http probe failed: 5, giveup 2024/11/19-15:16:34.269352 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00047/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 121.888 sec, real 11.516 (submit 424.518 ms wait 11.079 s), input 121.888 MiB, send shutdown 2024/11/19-15:16:34.276686 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-00 2024/11/19-15:16:34.276901 139851137550080 [dcompact_etcd.cc:1773] job-00047/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:34.311495 139851169007360 [dcompact_etcd.cc:1163] job-00054/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.309 sec, input = 244.368 MiB 2024/11/19-15:16:34.395894 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-02 2024/11/19-15:16:34.395973 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00053/att-02/compact.done) = No such file or directory: 0.659 sec on http probe failed: 4, retry 2024/11/19-15:16:34.560690 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-02 2024/11/19-15:16:34.560758 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00053/att-02/compact.done) = No such file or directory: 0.818 sec on http probe failed: 5, giveup 2024/11/19-15:16:34.560775 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00053/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 7.519 sec, real 1.127 (submit 295.180 ms wait 0.818 s), input 7.519 MiB, send shutdown 2024/11/19-15:16:34.562892 139851137550080 [dcompact_etcd.cc:1163] job-00047/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.286 sec, input = 121.888 MiB 2024/11/19-15:16:34.565671 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00053/att-02 2024/11/19-15:16:34.565885 139851127064320 [dcompact_etcd.cc:1773] job-00053/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:34.568024 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00053) = success 2024/11/19-15:16:34.568032 139851127064320 [dcompact_etcd.cc:1773] job-00053/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:16:34.568052 139851127064320 (Original Log Time 2024/11/19-15:16:31.154961) EVENT_LOG_v1 {"time_micros": 1732000591154954, "job": 53, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4429], "score": 2.42257, "input_data_size": 4034161, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:34.568059 139851127064320 [compaction_job.cc:2437] [default] [JOB 53] Compacting 1@2 files to L3, score 2.42, subcompactions 1 : 1 2024/11/19-15:16:34.568066 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 51 Base level 2, inputs: [4429(3939KB)] 2024/11/19-15:16:34.587187 139851127064320 [compaction_job.cc:703] job-00053: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:16:34.587680 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:16:34.587822 139851127064320 (Original Log Time 2024/11/19-15:16:34.568096) EVENT_LOG_v1 {"time_micros": 1732000594568077, "job": 53, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4429], "score": 2.42257, "input_data_size": 4034161, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:34.587833 139851127064320 (Original Log Time 2024/11/19-15:16:34.587252) [compaction_job.cc:2022] [default] [JOB 53] Compacted 1@2 files to L3 => 0 bytes 2024/11/19-15:16:34.587836 139851127064320 (Original Log Time 2024/11/19-15:16:34.587643) [compaction_job.cc:1224] [default] compacted to: files[5 51 67 75 0 0 0] max score 2.26 (101 files need compaction), MB/sec: 211.2 rd, 0.0 wr, level 3, files in(1, 0) out(0 +0 blob) MB in(3.8, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 180000, records dropped: 180000 output_compression: Snappy 2024/11/19-15:16:34.587852 139851127064320 (Original Log Time 2024/11/19-15:16:34.587670) EVENT_LOG_v1 {"time_micros": 1732000594587656, "cf": "default", "job": 53, "event": "compaction_finished", "compaction_time_micros": 19101, "compaction_time_cpu_micros": 19022, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 180000, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 51, 67, 75, 0, 0, 0]} 2024/11/19-15:16:34.587956 139851127064320 [compaction_job.cc:2437] [default] [JOB 55] Compacting 1@2 files to L3, score 2.26, subcompactions 1 : 1 2024/11/19-15:16:34.587964 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 53 Base level 2, inputs: [3940(62MB)] 2024/11/19-15:16:34.905937 139851127064320 [dcompact_etcd.cc:1163] job-00055/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.317 sec, input = 62.868 MiB 2024/11/19-15:16:35.711445 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-00 2024/11/19-15:16:35.711522 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00051/att-00/compact.done) = No such file or directory: 6.887 sec on http probe failed: 3, retry 2024/11/19-15:16:36.001768 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-01 2024/11/19-15:16:36.001845 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00044/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:16:36.483508 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-00 2024/11/19-15:16:36.483588 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00055/att-00/compact.done) = No such file or directory: 1.572 sec on http probe failed: 1, retry 2024/11/19-15:16:37.195161 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-01 2024/11/19-15:16:37.195252 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00045/att-01/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:16:37.271482 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-01 2024/11/19-15:16:37.271557 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00046/att-01/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:16:37.618830 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-01 2024/11/19-15:16:37.618941 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00047/att-01/compact.done) = No such file or directory: 3.047 sec on http probe failed: 1, retry 2024/11/19-15:16:37.719725 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-00 2024/11/19-15:16:37.719807 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00051/att-00/compact.done) = No such file or directory: 8.897 sec on http probe failed: 4, retry 2024/11/19-15:16:37.750239 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-00 2024/11/19-15:16:37.750303 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00055/att-00/compact.done) = No such file or directory: 2.835 sec on http probe failed: 2, retry 2024/11/19-15:16:38.011279 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-01 2024/11/19-15:16:38.011381 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00044/att-01/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:16:39.012685 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-00 2024/11/19-15:16:39.012761 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00055/att-00/compact.done) = No such file or directory: 4.102 sec on http probe failed: 3, retry 2024/11/19-15:16:39.203961 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-01 2024/11/19-15:16:39.204037 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00045/att-01/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:16:39.280088 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-01 2024/11/19-15:16:39.280150 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00046/att-01/compact.done) = No such file or directory: 5.177 sec on http probe failed: 2, retry 2024/11/19-15:16:39.321655 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-00 2024/11/19-15:16:39.321720 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00054/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:16:39.628105 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-01 2024/11/19-15:16:39.628225 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00047/att-01/compact.done) = No such file or directory: 5.056 sec on http probe failed: 2, retry 2024/11/19-15:16:39.730883 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-00 2024/11/19-15:16:39.730984 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00051/att-00/compact.done) = No such file or directory: 10.905 sec on http probe failed: 5, giveup 2024/11/19-15:16:39.731003 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00051/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 114.819 sec, real 11.209 (submit 292.297 ms wait 10.905 s), input 114.819 MiB, send shutdown 2024/11/19-15:16:39.735707 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-00 2024/11/19-15:16:39.735915 139851179493120 [dcompact_etcd.cc:1773] job-00051/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:40.028713 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-01 2024/11/19-15:16:40.028797 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00044/att-01/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:16:40.034349 139851179493120 [dcompact_etcd.cc:1163] job-00051/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.298 sec, input = 114.819 MiB 2024/11/19-15:16:40.279167 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-00 2024/11/19-15:16:40.279262 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00055/att-00/compact.done) = No such file or directory: 5.364 sec on http probe failed: 4, retry 2024/11/19-15:16:40.463008 139852362282752 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16273899 2024/11/19-15:16:41.212437 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-01 2024/11/19-15:16:41.212540 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00045/att-01/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:16:41.289336 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-01 2024/11/19-15:16:41.289435 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00046/att-01/compact.done) = No such file or directory: 7.185 sec on http probe failed: 3, retry 2024/11/19-15:16:41.329653 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-00 2024/11/19-15:16:41.329719 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00054/att-00/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:16:41.542388 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-00 2024/11/19-15:16:41.542462 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00055/att-00/compact.done) = No such file or directory: 6.631 sec on http probe failed: 5, giveup 2024/11/19-15:16:41.542480 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00055/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 62.868 sec, real 6.954 (submit 317.493 ms wait 6.631 s), input 62.868 MiB, send shutdown 2024/11/19-15:16:41.551497 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-00 2024/11/19-15:16:41.551714 139851127064320 [dcompact_etcd.cc:1773] job-00055/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:41.633771 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-01 2024/11/19-15:16:41.633851 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00047/att-01/compact.done) = No such file or directory: 7.065 sec on http probe failed: 3, retry 2024/11/19-15:16:41.827363 139851127064320 [dcompact_etcd.cc:1163] job-00055/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.276 sec, input = 62.868 MiB 2024/11/19-15:16:42.035411 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-01 2024/11/19-15:16:42.035504 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00044/att-01/compact.done) = No such file or directory: 9.209 sec on http probe failed: 4, retry 2024/11/19-15:16:42.910431 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-01 2024/11/19-15:16:42.910516 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00051/att-01/compact.done) = No such file or directory: 2.871 sec on http probe failed: 1, retry 2024/11/19-15:16:43.217834 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-01 2024/11/19-15:16:43.217906 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00045/att-01/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:16:43.297266 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-01 2024/11/19-15:16:43.297358 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00046/att-01/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:16:43.335037 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-00 2024/11/19-15:16:43.335116 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00054/att-00/compact.done) = No such file or directory: 9.018 sec on http probe failed: 3, retry 2024/11/19-15:16:43.407339 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-01 2024/11/19-15:16:43.407434 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00055/att-01/compact.done) = No such file or directory: 1.572 sec on http probe failed: 1, retry 2024/11/19-15:16:43.642250 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-01 2024/11/19-15:16:43.642338 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00047/att-01/compact.done) = No such file or directory: 9.071 sec on http probe failed: 4, retry 2024/11/19-15:16:44.043345 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-01 2024/11/19-15:16:44.043439 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00044/att-01/compact.done) = No such file or directory: 11.216 sec on http probe failed: 5, giveup 2024/11/19-15:16:44.043459 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00044/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.907 sec, real 11.508 (submit 284.316 ms wait 11.216 s), input 126.907 MiB, send shutdown 2024/11/19-15:16:44.052057 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-01 2024/11/19-15:16:44.052292 139850978166528 [dcompact_etcd.cc:1773] job-00044/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:44.339297 139850978166528 [dcompact_etcd.cc:1163] job-00044/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.287 sec, input = 126.907 MiB 2024/11/19-15:16:44.673757 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-01 2024/11/19-15:16:44.673849 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00055/att-01/compact.done) = No such file or directory: 2.838 sec on http probe failed: 2, retry 2024/11/19-15:16:44.915798 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-01 2024/11/19-15:16:44.915892 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00051/att-01/compact.done) = No such file or directory: 4.876 sec on http probe failed: 2, retry 2024/11/19-15:16:45.225292 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-01 2024/11/19-15:16:45.225380 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00045/att-01/compact.done) = No such file or directory: 11.203 sec on http probe failed: 5, giveup 2024/11/19-15:16:45.225398 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00045/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.825 sec, real 11.503 (submit 293.426 ms wait 11.203 s), input 126.825 MiB, send shutdown 2024/11/19-15:16:45.232916 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-01 2024/11/19-15:16:45.233146 139851148035840 [dcompact_etcd.cc:1773] job-00045/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:45.307826 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-01 2024/11/19-15:16:45.307897 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00046/att-01/compact.done) = No such file or directory: 11.203 sec on http probe failed: 5, giveup 2024/11/19-15:16:45.307914 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00046/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.836 sec, real 11.483 (submit 269.549 ms wait 11.203 s), input 126.836 MiB, send shutdown 2024/11/19-15:16:45.316665 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-01 2024/11/19-15:16:45.316872 139851158521600 [dcompact_etcd.cc:1773] job-00046/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:45.343578 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-00 2024/11/19-15:16:45.343644 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00054/att-00/compact.done) = No such file or directory: 11.024 sec on http probe failed: 4, retry 2024/11/19-15:16:45.513911 139851148035840 [dcompact_etcd.cc:1163] job-00045/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 126.825 MiB 2024/11/19-15:16:45.621653 139851158521600 [dcompact_etcd.cc:1163] job-00046/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.305 sec, input = 126.836 MiB 2024/11/19-15:16:45.647563 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-01 2024/11/19-15:16:45.647627 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00047/att-01/compact.done) = No such file or directory: 11.080 sec on http probe failed: 5, giveup 2024/11/19-15:16:45.647645 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00047/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 121.888 sec, real 11.371 (submit 285.911 ms wait 11.080 s), input 121.888 MiB, send shutdown 2024/11/19-15:16:45.655620 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-01 2024/11/19-15:16:45.655833 139851137550080 [dcompact_etcd.cc:1773] job-00047/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:45.936415 139851137550080 [dcompact_etcd.cc:1163] job-00047/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 121.888 MiB 2024/11/19-15:16:45.940760 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-01 2024/11/19-15:16:45.940840 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00055/att-01/compact.done) = No such file or directory: 4.104 sec on http probe failed: 3, retry 2024/11/19-15:16:46.924299 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-01 2024/11/19-15:16:46.924390 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00051/att-01/compact.done) = No such file or directory: 6.882 sec on http probe failed: 3, retry 2024/11/19-15:16:47.207196 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-01 2024/11/19-15:16:47.207314 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00055/att-01/compact.done) = No such file or directory: 5.371 sec on http probe failed: 4, retry 2024/11/19-15:16:47.348382 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-00 2024/11/19-15:16:47.348462 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00054/att-00/compact.done) = No such file or directory: 13.032 sec on http probe failed: 5, giveup 2024/11/19-15:16:47.348481 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00054/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 244.368 sec, real 13.346 (submit 308.961 ms wait 13.032 s), input 244.368 MiB, send shutdown 2024/11/19-15:16:47.356965 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-00 2024/11/19-15:16:47.357210 139851169007360 [dcompact_etcd.cc:1773] job-00054/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:47.518551 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-02 2024/11/19-15:16:47.518667 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00044/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:16:47.653615 139851169007360 [dcompact_etcd.cc:1163] job-00054/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.296 sec, input = 244.368 MiB 2024/11/19-15:16:48.473322 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-01 2024/11/19-15:16:48.473400 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00055/att-01/compact.done) = No such file or directory: 6.637 sec on http probe failed: 5, giveup 2024/11/19-15:16:48.473419 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00055/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 62.868 sec, real 6.922 (submit 275.556 ms wait 6.637 s), input 62.868 MiB, send shutdown 2024/11/19-15:16:48.478960 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-01 2024/11/19-15:16:48.479173 139851127064320 [dcompact_etcd.cc:1773] job-00055/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:48.694216 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-02 2024/11/19-15:16:48.694308 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00045/att-02/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:16:48.707572 139834427442944 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777215 2024/11/19-15:16:48.747173 139851127064320 [dcompact_etcd.cc:1163] job-00055/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.268 sec, input = 62.868 MiB 2024/11/19-15:16:48.802808 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-02 2024/11/19-15:16:48.802875 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00046/att-02/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:16:48.934939 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-01 2024/11/19-15:16:48.934996 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00051/att-01/compact.done) = No such file or directory: 8.890 sec on http probe failed: 4, retry 2024/11/19-15:16:48.993246 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-02 2024/11/19-15:16:48.993309 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00047/att-02/compact.done) = No such file or directory: 3.047 sec on http probe failed: 1, retry 2024/11/19-15:16:49.527795 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-02 2024/11/19-15:16:49.527887 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00044/att-02/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:16:50.327801 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-02 2024/11/19-15:16:50.327881 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00055/att-02/compact.done) = No such file or directory: 1.572 sec on http probe failed: 1, retry 2024/11/19-15:16:50.704499 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-02 2024/11/19-15:16:50.704593 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00045/att-02/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:16:50.812412 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-02 2024/11/19-15:16:50.812512 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00046/att-02/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:16:50.942998 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-01 2024/11/19-15:16:50.943085 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00051/att-01/compact.done) = No such file or directory: 10.901 sec on http probe failed: 5, giveup 2024/11/19-15:16:50.943103 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00051/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 114.819 sec, real 11.207 (submit 298.336 ms wait 10.901 s), input 114.819 MiB, send shutdown 2024/11/19-15:16:50.952076 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-01 2024/11/19-15:16:50.952287 139851179493120 [dcompact_etcd.cc:1773] job-00051/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:50.999499 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-02 2024/11/19-15:16:50.999560 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00047/att-02/compact.done) = No such file or directory: 5.057 sec on http probe failed: 2, retry 2024/11/19-15:16:51.220236 139851179493120 [dcompact_etcd.cc:1163] job-00051/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.268 sec, input = 114.819 MiB 2024/11/19-15:16:51.533996 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-02 2024/11/19-15:16:51.534088 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00044/att-02/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:16:51.593299 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-02 2024/11/19-15:16:51.593361 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00055/att-02/compact.done) = No such file or directory: 2.838 sec on http probe failed: 2, retry 2024/11/19-15:16:52.659901 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-01 2024/11/19-15:16:52.660019 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00054/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:16:52.710545 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-02 2024/11/19-15:16:52.710624 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00045/att-02/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:16:52.820259 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-02 2024/11/19-15:16:52.820337 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00046/att-02/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:16:52.859499 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-02 2024/11/19-15:16:52.859562 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00055/att-02/compact.done) = No such file or directory: 4.104 sec on http probe failed: 3, retry 2024/11/19-15:16:53.010172 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-02 2024/11/19-15:16:53.010236 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00047/att-02/compact.done) = No such file or directory: 7.063 sec on http probe failed: 3, retry 2024/11/19-15:16:53.410811 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4443. Immutable memtables: 0. 2024/11/19-15:16:53.411256 139852362282752 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16273898 2024/11/19-15:16:53.411354 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 57] Syncing log #4435 2024/11/19-15:16:53.411441 139850957195008 (Original Log Time 2024/11/19-15:16:53.411340) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:16:53.411451 139850957195008 [flush_job.cc:879] [default] [JOB 57] Flushing memtable with next log file: 4443 2024/11/19-15:16:53.411471 139850957195008 EVENT_LOG_v1 {"time_micros": 1732000613411466, "job": 57, "event": "flush_started", "num_memtables": 1, "num_entries": 1219391, "num_deletes": 0, "total_data_size": 476636612, "memory_usage": 544210312, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:16:53.411475 139850957195008 [flush_job.cc:910] [default] [JOB 57] Level-0 flush table #4444: started 2024/11/19-15:16:53.411798 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000005.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004444.sst in ConvertToSST, mem_size = 518.999 M, time = 314.548 us 2024/11/19-15:16:53.547892 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-02 2024/11/19-15:16:53.547994 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00044/att-02/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:16:53.957701 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004444.sst): fsize = 519.001 M, time(ms): open: 0.039, seek: 0.026, finish: 0.034, meta: 0.030, Flush: 0.012, sync: 545.730, close: 0.015, all: 545.886 2024/11/19-15:16:53.958865 139850957195008 [flush_job.cc:1083] [default] [JOB 57] Flush lasted 547423 microseconds, and 323441 cpu microseconds. 2024/11/19-15:16:53.980352 139850957195008 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 15785681 2024/11/19-15:16:53.980445 139850957195008 (Original Log Time 2024/11/19-15:16:53.957756) [flush_job.cc:1029] [default] [JOB 57] Level-0 flush table #4444: 544211738 bytes OK (needs compaction) 2024/11/19-15:16:53.980451 139850957195008 (Original Log Time 2024/11/19-15:16:53.958881) [memtable_list.cc:560] [default] Level-0 commit table #4444 started 2024/11/19-15:16:53.980455 139850957195008 (Original Log Time 2024/11/19-15:16:53.980321) [memtable_list.cc:759] [default] Level-0 commit table #4444: memtable #1 done 2024/11/19-15:16:53.980457 139850957195008 (Original Log Time 2024/11/19-15:16:53.980344) EVENT_LOG_v1 {"time_micros": 1732000613980339, "job": 57, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [6, 16, 10, 48, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:16:53.980460 139850957195008 (Original Log Time 2024/11/19-15:16:53.980418) [db_impl_compaction_flush.cc:310] [default] Level summary: files[6 16 10 48 0 0 0] max score 6.03 (9 files need compaction) 2024/11/19-15:16:54.096023 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-02 2024/11/19-15:16:54.096099 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00051/att-02/compact.done) = No such file or directory: 2.871 sec on http probe failed: 1, retry 2024/11/19-15:16:54.121353 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-02 2024/11/19-15:16:54.121430 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00055/att-02/compact.done) = No such file or directory: 5.370 sec on http probe failed: 4, retry 2024/11/19-15:16:54.664956 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-01 2024/11/19-15:16:54.665042 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00054/att-01/compact.done) = No such file or directory: 7.006 sec on http probe failed: 2, retry 2024/11/19-15:16:54.715799 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-02 2024/11/19-15:16:54.715865 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00045/att-02/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:16:54.825816 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-02 2024/11/19-15:16:54.825880 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00046/att-02/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:16:55.018397 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-02 2024/11/19-15:16:55.018493 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00047/att-02/compact.done) = No such file or directory: 9.074 sec on http probe failed: 4, retry 2024/11/19-15:16:55.388008 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-02 2024/11/19-15:16:55.388105 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00055/att-02/compact.done) = No such file or directory: 6.632 sec on http probe failed: 5, giveup 2024/11/19-15:16:55.388123 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00055/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 62.868 sec, real 6.909 (submit 267.914 ms wait 6.632 s), input 62.868 MiB, send shutdown 2024/11/19-15:16:55.395587 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00055/att-02 2024/11/19-15:16:55.395845 139851127064320 [dcompact_etcd.cc:1773] job-00055/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:55.397608 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00055) = success 2024/11/19-15:16:55.397617 139851127064320 [dcompact_etcd.cc:1773] job-00055/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:16:55.397639 139851127064320 (Original Log Time 2024/11/19-15:16:34.587977) EVENT_LOG_v1 {"time_micros": 1732000594587971, "job": 55, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [3940], "score": 2.25858, "input_data_size": 65126289, "oldest_snapshot_seqno": 112846808} 2024/11/19-15:16:55.397647 139851127064320 [compaction_job.cc:2437] [default] [JOB 55] Compacting 1@2 files to L3, score 2.26, subcompactions 1 : 1 2024/11/19-15:16:55.397654 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 53 Base level 2, inputs: [3940(62MB)] 2024/11/19-15:16:55.404812 139851127064320 [compaction_job.cc:703] job-00055: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:16:55.405094 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16273897 2024/11/19-15:16:55.405206 139851127064320 (Original Log Time 2024/11/19-15:16:55.397682) EVENT_LOG_v1 {"time_micros": 1732000615397666, "job": 55, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [3940], "score": 2.25858, "input_data_size": 65126289, "oldest_snapshot_seqno": 112846808} 2024/11/19-15:16:55.405212 139851127064320 (Original Log Time 2024/11/19-15:16:55.404869) [compaction_job.cc:2022] [default] [JOB 55] Compacted 1@2 files to L3 => 0 bytes 2024/11/19-15:16:55.405215 139851127064320 (Original Log Time 2024/11/19-15:16:55.405065) [compaction_job.cc:1224] [default] compacted to: files[6 16 9 48 0 0 0] max score 6.03 (9 files need compaction), MB/sec: 9116.2 rd, 0.0 wr, level 3, files in(1, 0) out(0 +0 blob) MB in(62.1, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 63572, records dropped: 63572 output_compression: Snappy 2024/11/19-15:16:55.405234 139851127064320 (Original Log Time 2024/11/19-15:16:55.405088) EVENT_LOG_v1 {"time_micros": 1732000615405078, "cf": "default", "job": 55, "event": "compaction_finished", "compaction_time_micros": 7144, "compaction_time_cpu_micros": 7086, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 63572, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 16, 9, 48, 0, 0, 0]} 2024/11/19-15:16:55.405680 139851127064320 [compaction_job.cc:2437] [default] [JOB 58] Compacting 1@0 files to L1, score 6.03, subcompactions 3 : 3 2024/11/19-15:16:55.405689 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 56 Base level 0, inputs: [4325(518MB)] 2024/11/19-15:16:55.406344 139851127064320 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4445: 4 keys, 3396 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.406398 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000615406368, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4445, "file_size": 3396, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 128, "index_size": 2432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 156, "raw_average_key_size": 39, "raw_value_size": 80, "raw_average_value_size": 20, "num_data_blocks": 1, "num_entries": 4, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 39, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4445, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.409237 139851127064320 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4448: 4 keys, 3588 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.409288 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000615409260, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4448, "file_size": 3588, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 512, "index_size": 2240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 64, "raw_average_key_size": 16, "raw_value_size": 461, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 4, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4448, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.410009 139851127064320 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4449: 20 keys, 6469 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.410054 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000615410029, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4449, "file_size": 6469, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 2432, "index_size": 3200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 400, "raw_average_key_size": 20, "raw_value_size": 2413, "raw_average_value_size": 120, "num_data_blocks": 1, "num_entries": 20, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4449, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.411274 139851127064320 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4450: 457 keys, 451660 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.411320 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000615411294, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4450, "file_size": 451660, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 294464, "index_size": 156352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 10968, "raw_average_key_size": 24, "raw_value_size": 294412, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 457, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4450, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.414693 139849937979136 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4446: 21512 keys, 1086094 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.414744 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000615414722, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4446, "file_size": 1086094, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 424512, "index_size": 660736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 516288, "raw_average_key_size": 24, "raw_value_size": 424504, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 21512, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4446, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.444039 139833664079616 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4447: 49903 keys, 33507026 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.444107 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000615444085, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4447, "file_size": 33507026, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32158080, "index_size": 1348096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1197672, "raw_average_key_size": 24, "raw_value_size": 32158030, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49903, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4447, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.451729 139833664079616 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4452: 9640 keys, 6601809 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.451774 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000615451754, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4452, "file_size": 6601809, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 6227968, "index_size": 372992, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 231360, "raw_average_key_size": 24, "raw_value_size": 6227925, "raw_average_value_size": 646, "num_data_blocks": 1, "num_entries": 9640, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4452, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.474797 139833664079616 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4453: 60000 keys, 4834386 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.474864 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000615474835, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4453, "file_size": 4834386, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 3240384, "index_size": 1593152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1200000, "raw_average_key_size": 20, "raw_value_size": 3240348, "raw_average_value_size": 54, "num_data_blocks": 1, "num_entries": 60000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4453, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.487520 139833664079616 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4454: 18000 keys, 552284 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.487584 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000615487554, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4454, "file_size": 552284, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 18048, "index_size": 533376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 432000, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 18000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4454, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.507906 139833664079616 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4455: 38488 keys, 1836430 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.507966 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000615507945, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4455, "file_size": 1836430, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 763520, "index_size": 1072064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 923712, "raw_average_key_size": 24, "raw_value_size": 763496, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 38488, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4455, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.520758 139849937979136 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4451: 231249 keys, 33325269 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.520804 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000615520784, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4451, "file_size": 33325269, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27034048, "index_size": 6290368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6474972, "raw_average_key_size": 28, "raw_value_size": 27033994, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231249, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4451, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.557478 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-02 2024/11/19-15:16:55.557570 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00044/att-02/compact.done) = No such file or directory: 11.209 sec on http probe failed: 5, giveup 2024/11/19-15:16:55.557588 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00044/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.907 sec, real 11.505 (submit 286.923 ms wait 11.209 s), input 126.907 MiB, send shutdown 2024/11/19-15:16:55.562010 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00044/att-02 2024/11/19-15:16:55.562205 139850978166528 [dcompact_etcd.cc:1773] job-00044/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:55.564051 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00044) = success 2024/11/19-15:16:55.564061 139850978166528 [dcompact_etcd.cc:1773] job-00044/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:16:55.564081 139850978166528 (Original Log Time 2024/11/19-15:16:21.007328) EVENT_LOG_v1 {"time_micros": 1732000581007322, "job": 44, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4402, 4403, 4405, 4407], "score": 3.15389, "input_data_size": 134098240, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:55.564087 139850978166528 [compaction_job.cc:2437] [default] [JOB 44] Compacting 4@1 files to L2, score 3.15, subcompactions 1 : 1 2024/11/19-15:16:55.564095 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 42 Base level 1, inputs: [4402(31MB) 4403(31MB) 4405(31MB) 4407(31MB)] 2024/11/19-15:16:55.564309 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004457.sst) /storage/mytopling/..//tmp/Topling-82038-wVfqvK 2024/11/19-15:16:55.618049 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004457.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:16:55.621361 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004457.sst): index pass time = 0.00's, 157.984'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64714 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:16:55.623263 139849937979136 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4456: 231473 keys, 33325077 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.623314 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000615623291, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4456, "file_size": 33325077, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27025536, "index_size": 6298688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481244, "raw_average_key_size": 28, "raw_value_size": 27025522, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231473, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4456, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.682424 139849937979136 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4458: 136865 keys, 19752021 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.682480 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000615682458, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4458, "file_size": 19752021, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 15971200, "index_size": 3779968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3832220, "raw_average_key_size": 28, "raw_value_size": 15971170, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 136865, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4458, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.684967 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004457.sst): first pass time = 0.12's, 556.246'MB/sec 2024/11/19-15:16:55.685017 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004457.sst): old prealloc_size = 73819750, real_size = 66151904 2024/11/19-15:16:55.685073 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004457.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64714 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:16:55.685113 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004457.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.067 GB 2024/11/19-15:16:55.716363 139849937979136 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4459: 100000 keys, 9267540 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.716419 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000615716398, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4459, "file_size": 9267540, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 6701312, "index_size": 2565376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1600000, "raw_average_key_size": 16, "raw_value_size": 6701254, "raw_average_value_size": 67, "num_data_blocks": 1, "num_entries": 100000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4459, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.748025 139849937979136 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4460: 32060 keys, 33525202 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.748082 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000615748060, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4460, "file_size": 33525202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32605312, "index_size": 919040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32605259, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4460, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.750566 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004457.sst): second pass time = 0.00's,220855114.094'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 1009.363'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2153.720'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.372'MB/sec write SST data time = 0.02's, 3015.144'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 220855114.094'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64714 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 64714 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0464 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 2.004956 GB avg = 0.380 KB (by entry num) total key len = 0.132749 GB avg = 0.025 KB total ukey len = 0.090563 GB avg = 0.017 KB total ukey num = 0.005273310 Billion total entry num = 0.005273310 Billion write speed all = 17.664081879 MB/sec (with seq num) write speed all = 17.315490638 MB/sec (without seq num) 2024/11/19-15:16:55.755795 139850978166528 [compaction_job.cc:1948] [default] [JOB 44] Generated table #4457: 64714 keys, 66137133 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.755873 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000615755838, "cf_name": "default", "job": 44, "event": "table_file_creation", "file_number": 4457, "file_size": 66137133, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025360, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294280, "raw_average_key_size": 20, "raw_value_size": 65814824, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64714, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4457, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:55.774463 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004462.sst) /storage/mytopling/..//tmp/Topling-82038-5fzv1Z 2024/11/19-15:16:55.779897 139849937979136 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4461: 32107 keys, 33525138 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.779950 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000615779929, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4461, "file_size": 33525138, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32654144, "index_size": 870144, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642140, "raw_average_key_size": 20, "raw_value_size": 32654096, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32107, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4461, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.811701 139849937979136 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4463: 32057 keys, 33524690 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.811752 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000615811731, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4463, "file_size": 33524690, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32602816, "index_size": 921024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602805, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4463, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.827311 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004462.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:16:55.830540 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004462.sst): index pass time = 0.00's, 158.959'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63611 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:16:55.843445 139849937979136 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4464: 32057 keys, 33524690 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.843492 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000615843471, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4464, "file_size": 33524690, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32603200, "index_size": 920640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32603167, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4464, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.874864 139849937979136 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4465: 32151 keys, 33525138 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.874934 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000615874902, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4465, "file_size": 33525138, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32696704, "index_size": 827584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32696694, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4465, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.893796 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004462.sst): first pass time = 0.12's, 552.850'MB/sec 2024/11/19-15:16:55.893848 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004462.sst): old prealloc_size = 73819750, real_size = 65022000 2024/11/19-15:16:55.893914 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004462.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63611 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1016.97 avg-zval =1020.23 2024/11/19-15:16:55.893965 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004462.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.066 GB 2024/11/19-15:16:55.906260 139849937979136 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4466: 32057 keys, 33524626 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.906310 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000615906289, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4466, "file_size": 33524626, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32602048, "index_size": 921728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601993, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4466, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.938085 139849937979136 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4467: 32060 keys, 33525202 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.938140 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000615938119, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4467, "file_size": 33525202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32603968, "index_size": 920384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32603958, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4467, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.953292 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004462.sst): second pass time = 0.00's,182227090.141'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1093.878'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1509.312'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.368'MB/sec write SST data time = 0.02's, 4016.422'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 182227090.141'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63611 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.97 avg-zval = 1020.23 usrkeys = 63611 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.97 avg-zval = 1020.23 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7927 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0449 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 2.069647 GB avg = 0.388 KB (by entry num) total key len = 0.134022 GB avg = 0.025 KB total ukey len = 0.091326 GB avg = 0.017 KB total ukey num = 0.005336921 Billion total entry num = 0.005336921 Billion write speed all = 18.178686194 MB/sec (with seq num) write speed all = 17.826479984 MB/sec (without seq num) 2024/11/19-15:16:55.957001 139850978166528 [compaction_job.cc:1948] [default] [JOB 44] Generated table #4462: 63611 keys, 65007469 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.957050 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000615957028, "cf_name": "default", "job": 44, "event": "table_file_creation", "file_number": 4462, "file_size": 65007469, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64897568, "index_size": 108352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1272220, "raw_average_key_size": 20, "raw_value_size": 64690617, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63611, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4462, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:55.971993 139850978166528 [compaction_job.cc:703] job-00044: subcompact[0], size: 0.131145 G, files: /storage/mytopling/.rocksdb [004457,004462] 2024/11/19-15:16:55.972635 139850978166528 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777213 2024/11/19-15:16:55.972771 139850978166528 (Original Log Time 2024/11/19-15:16:55.564128) EVENT_LOG_v1 {"time_micros": 1732000615564107, "job": 44, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4402, 4403, 4405, 4407], "score": 3.15389, "input_data_size": 134098240, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:55.972778 139850978166528 (Original Log Time 2024/11/19-15:16:55.972425) [compaction_job.cc:2022] [default] [JOB 44] Compacted 4@1 files to L2 => 131144602 bytes 2024/11/19-15:16:55.972781 139850978166528 (Original Log Time 2024/11/19-15:16:55.972605) [compaction_job.cc:1224] [default] compacted to: files[6 12 11 48 0 0 0] max score 5.02 (10 files need compaction), MB/sec: 328.8 rd, 321.5 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.1 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128325, records dropped: 0 output_compression: Snappy 2024/11/19-15:16:55.972799 139850978166528 (Original Log Time 2024/11/19-15:16:55.972629) EVENT_LOG_v1 {"time_micros": 1732000615972617, "cf": "default", "job": 44, "event": "compaction_finished", "compaction_time_micros": 407878, "compaction_time_cpu_micros": 405900, "output_level": 2, "num_output_files": 2, "total_output_size": 131144602, "num_input_records": 128325, "num_output_records": 128325, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 12, 11, 48, 0, 0, 0]} 2024/11/19-15:16:55.976493 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004293.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:55.976519 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000615976517, "job": 44, "event": "table_file_deletion", "file_number": 4293} 2024/11/19-15:16:55.981304 139849937979136 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4468: 32152 keys, 33525074 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:55.981354 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000615981328, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4468, "file_size": 33525074, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32697664, "index_size": 826560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643040, "raw_average_key_size": 20, "raw_value_size": 32697630, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32152, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4468, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:55.983581 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004304.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:55.983603 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000615983601, "job": 44, "event": "table_file_deletion", "file_number": 4304} 2024/11/19-15:16:55.990930 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004305.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:55.990951 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000615990949, "job": 44, "event": "table_file_deletion", "file_number": 4305} 2024/11/19-15:16:55.997928 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004312.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:55.997948 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000615997946, "job": 44, "event": "table_file_deletion", "file_number": 4312} 2024/11/19-15:16:55.998083 139850978166528 [compaction_job.cc:2437] [default] [JOB 59] Compacting 1@2 + 2@3 files to L3, score 0.31, subcompactions 1 : 1 2024/11/19-15:16:55.998093 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 57 Base level 2, inputs: [4081(3910B)], [3816(2234B) 3952(3067B)] 2024/11/19-15:16:56.014022 139849937979136 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4469: 32059 keys, 33524754 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.014066 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000616014046, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4469, "file_size": 33524754, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32603776, "index_size": 920128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603717, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000615, "oldest_key_time": 0, "file_creation_time": 1732000615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4469, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.046027 139849937979136 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4470: 32104 keys, 33524754 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.046075 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000616046054, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4470, "file_size": 33524754, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32650496, "index_size": 873408, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642080, "raw_average_key_size": 20, "raw_value_size": 32650496, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32104, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4470, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.055392 139849937979136 [compaction_job.cc:1948] [default] [JOB 58] Generated table #4471: 9136 keys, 9541457 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.055433 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000616055414, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 4471, "file_size": 9541457, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 9290816, "index_size": 249792, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 182720, "raw_average_key_size": 20, "raw_value_size": 9290759, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 9136, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4471, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.055516 139851127064320 [compaction_job.cc:703] job-00058: subcompact[0], size: 0.000465 G, files: /storage/mytopling/.rocksdb [004445,004448,004449,004450] 2024/11/19-15:16:56.055532 139851127064320 [compaction_job.cc:703] job-00058: subcompact[1], size: 0.047332 G, files: /storage/mytopling/.rocksdb [004447,004452,004453,004454,004455] 2024/11/19-15:16:56.055539 139851127064320 [compaction_job.cc:703] job-00058: subcompact[2], size: 0.441547 G, files: /storage/mytopling/.rocksdb [004446,004451,004456,004458,004459,004460,004461,004463,004464,004465,004466,004467,004468,004469,004470,004471] 2024/11/19-15:16:56.072531 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:16:56.072662 139851127064320 (Original Log Time 2024/11/19-15:16:55.405704) EVENT_LOG_v1 {"time_micros": 1732000615405696, "job": 58, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [4325], "score": 6.03468, "input_data_size": 543425306, "oldest_snapshot_seqno": 113854381} 2024/11/19-15:16:56.072669 139851127064320 (Original Log Time 2024/11/19-15:16:56.072250) [compaction_job.cc:2022] [default] [JOB 58] Compacted 1@0 files to L1 => 489343774 bytes 2024/11/19-15:16:56.072672 139851127064320 (Original Log Time 2024/11/19-15:16:56.072498) [compaction_job.cc:1224] [default] compacted to: files[5 37 11 48 0 0 0] max score 3.04 (34 files need compaction), MB/sec: 836.2 rd, 753.0 wr, level 1, files in(1, 0) out(25 +0 blob) MB in(518.3, 0.0 +0.0 blob) out(466.7 +0.0 blob), read-write-amplify(1.9) write-amplify(0.9) OK, records in: 1227615, records dropped: 0 output_compression: Snappy 2024/11/19-15:16:56.072684 139851127064320 (Original Log Time 2024/11/19-15:16:56.072524) EVENT_LOG_v1 {"time_micros": 1732000616072509, "cf": "default", "job": 58, "event": "compaction_finished", "compaction_time_micros": 649839, "compaction_time_cpu_micros": 747841, "output_level": 1, "num_output_files": 25, "total_output_size": 489343774, "num_input_records": 1227615, "num_output_records": 1227615, "num_subcompactions": 3, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 37, 11, 48, 0, 0, 0]} 2024/11/19-15:16:56.081476 139851127064320 [compaction_job.cc:2437] [default] [JOB 60] Compacting 5@0 + 23@1 files to L1, score 3.04, subcompactions 3 : 3 2024/11/19-15:16:56.081495 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 58 Base level 0, inputs: [4444(519MB) 4436(512MB) 4432(512MB) 4428(514MB) 4340(513MB)], [4449(6469B) 4450(441KB) 4447(31MB) 4452(6447KB) 4453(4721KB) 4454(539KB) 4455(1793KB) 4446(1060KB) 4451(31MB) 4456(31MB) 4458(18MB) 4459(9050KB) 4460(31MB) 4461(31MB) 4463(31MB) 4464(31MB) 4465(31MB) 4466(31MB) 4467(31MB) 4468(31MB) 4469(31MB) 4470(31MB) 4471(9317KB)] 2024/11/19-15:16:56.082192 139851127064320 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4473: 14 keys, 5573 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.082243 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000616082215, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4473, "file_size": 5573, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 1664, "index_size": 3072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 224, "raw_average_key_size": 16, "raw_value_size": 1610, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 14, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4473, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.082961 139851127064320 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4475: 160 keys, 27658 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.083009 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000616082981, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4475, "file_size": 27658, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 19328, "index_size": 7488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3200, "raw_average_key_size": 20, "raw_value_size": 19284, "raw_average_value_size": 120, "num_data_blocks": 1, "num_entries": 160, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4475, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.101240 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-02 2024/11/19-15:16:56.101349 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00051/att-02/compact.done) = No such file or directory: 4.876 sec on http probe failed: 2, retry 2024/11/19-15:16:56.135901 139851127064320 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4476: 49900 keys, 33506962 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.135967 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000616135945, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4476, "file_size": 33506962, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32157120, "index_size": 1348992, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1197600, "raw_average_key_size": 24, "raw_value_size": 32157105, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49900, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4476, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.174352 139851127064320 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4477: 49835 keys, 33507090 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.174407 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000616174385, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4477, "file_size": 33507090, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32158080, "index_size": 1348160, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196040, "raw_average_key_size": 24, "raw_value_size": 32158069, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49835, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4477, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.205901 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4474: 231247 keys, 33325205 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.205986 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000616205963, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4474, "file_size": 33325205, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27032768, "index_size": 6291584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6474916, "raw_average_key_size": 28, "raw_value_size": 27032730, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231247, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4474, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.212142 139851127064320 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4478: 49882 keys, 33506898 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.212186 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000616212166, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4478, "file_size": 33506898, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32158464, "index_size": 1347584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1197168, "raw_average_key_size": 24, "raw_value_size": 32158428, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49882, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4478, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.230836 139833664079616 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4472: 419514 keys, 32925205 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.230905 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000616230884, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4472, "file_size": 32925205, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 22655168, "index_size": 10269184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 8390280, "raw_average_key_size": 20, "raw_value_size": 22655161, "raw_average_value_size": 54, "num_data_blocks": 1, "num_entries": 419514, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4472, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.250376 139851127064320 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4480: 49859 keys, 33507090 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.250420 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000616250401, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4480, "file_size": 33507090, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32158464, "index_size": 1347776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196616, "raw_average_key_size": 24, "raw_value_size": 32158428, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49859, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4480, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.286612 139833664079616 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4481: 144000 keys, 3763296 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.286665 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000616286644, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4481, "file_size": 3763296, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 144000, "index_size": 3618432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3456000, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 144000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4481, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.287521 139851127064320 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4482: 49820 keys, 33506514 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.287565 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000616287545, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4482, "file_size": 33506514, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32156032, "index_size": 1349632, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1195680, "raw_average_key_size": 24, "raw_value_size": 32155973, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49820, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4482, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.288585 139850978166528 [dcompact_etcd.cc:1163] job-00059/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.290 sec, input = 4.762 KiB 2024/11/19-15:16:56.296593 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-00 2024/11/19-15:16:56.296667 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00059/att-00/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:16:56.301366 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-00 2024/11/19-15:16:56.301428 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00059/att-00/compact.done) = No such file or directory: 0.008 sec on http probe failed: 2, retry 2024/11/19-15:16:56.306076 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-00 2024/11/19-15:16:56.306139 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00059/att-00/compact.done) = No such file or directory: 0.013 sec on http probe failed: 3, retry 2024/11/19-15:16:56.308612 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4479: 231453 keys, 33325077 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.308656 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000616308635, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4479, "file_size": 33325077, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27026944, "index_size": 6297280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480684, "raw_average_key_size": 28, "raw_value_size": 27026930, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231453, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4479, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.315799 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-00 2024/11/19-15:16:56.315865 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00059/att-00/compact.done) = No such file or directory: 0.018 sec on http probe failed: 4, retry 2024/11/19-15:16:56.320487 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-00 2024/11/19-15:16:56.320546 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00059/att-00/compact.done) = No such file or directory: 0.027 sec on http probe failed: 5, giveup 2024/11/19-15:16:56.320566 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00059/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.322 (submit 290.033 ms wait 0.027 s), input 4.762 KiB, send shutdown 2024/11/19-15:16:56.325330 139851127064320 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4484: 49853 keys, 33507218 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.325372 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000616325352, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4484, "file_size": 33507218, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32158592, "index_size": 1347776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196472, "raw_average_key_size": 24, "raw_value_size": 32158544, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49853, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4484, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.328322 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-00 2024/11/19-15:16:56.328530 139850978166528 [dcompact_etcd.cc:1773] job-00059/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:56.363760 139851127064320 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4486: 49866 keys, 33506514 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.363807 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000616363786, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4486, "file_size": 33506514, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32157952, "index_size": 1347712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196784, "raw_average_key_size": 24, "raw_value_size": 32157916, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49866, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4486, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.401915 139851127064320 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4487: 49878 keys, 33506898 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.401972 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000616401950, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4487, "file_size": 33506898, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32157760, "index_size": 1348288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1197072, "raw_average_key_size": 24, "raw_value_size": 32157758, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49878, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4487, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.412121 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4485: 231467 keys, 33324949 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.412169 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000616412148, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4485, "file_size": 33324949, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27023744, "index_size": 6300352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481076, "raw_average_key_size": 28, "raw_value_size": 27023738, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231467, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4485, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.439500 139851127064320 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4488: 49856 keys, 33507218 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.439546 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000616439525, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4488, "file_size": 33507218, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32159040, "index_size": 1347328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196544, "raw_average_key_size": 24, "raw_value_size": 32158998, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49856, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4488, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.463070 139851127064320 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4490: 31251 keys, 21065618 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.463118 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000616463097, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4490, "file_size": 21065618, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 20167936, "index_size": 896832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 750024, "raw_average_key_size": 24, "raw_value_size": 20167916, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 31251, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4490, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.463744 139833664079616 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4483: 480000 keys, 21273877 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.463786 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000616463766, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4483, "file_size": 21273877, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 9504000, "index_size": 11769024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 11520000, "raw_average_key_size": 24, "raw_value_size": 9504000, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 480000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4483, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.483455 139851127064320 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4491: 60486 keys, 4872402 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.483493 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000616483476, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4491, "file_size": 4872402, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 3266624, "index_size": 1604928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1209720, "raw_average_key_size": 20, "raw_value_size": 3266582, "raw_average_value_size": 54, "num_data_blocks": 1, "num_entries": 60486, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4491, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.513073 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4489: 231380 keys, 33325077 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.513113 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000616513094, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4489, "file_size": 33325077, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27030336, "index_size": 6293888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6478640, "raw_average_key_size": 28, "raw_value_size": 27030280, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231380, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4489, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.560781 139833664079616 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4492: 231249 keys, 33325269 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.560824 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000616560804, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4492, "file_size": 33325269, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27034048, "index_size": 6290368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6474972, "raw_average_key_size": 28, "raw_value_size": 27033994, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231249, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4492, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.608133 139850978166528 [dcompact_etcd.cc:1163] job-00059/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 4.762 KiB 2024/11/19-15:16:56.613464 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4493: 231368 keys, 33325077 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.613505 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000616613486, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4493, "file_size": 33325077, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27031104, "index_size": 6293120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6478304, "raw_average_key_size": 28, "raw_value_size": 27031064, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231368, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4493, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.613543 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-01 2024/11/19-15:16:56.613617 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00059/att-01/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:16:56.622605 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-01 2024/11/19-15:16:56.622672 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00059/att-01/compact.done) = No such file or directory: 0.006 sec on http probe failed: 2, retry 2024/11/19-15:16:56.628302 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-01 2024/11/19-15:16:56.628376 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00059/att-01/compact.done) = No such file or directory: 0.015 sec on http probe failed: 3, retry 2024/11/19-15:16:56.637468 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-01 2024/11/19-15:16:56.637542 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00059/att-01/compact.done) = No such file or directory: 0.020 sec on http probe failed: 4, retry 2024/11/19-15:16:56.644987 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-01 2024/11/19-15:16:56.645055 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00059/att-01/compact.done) = No such file or directory: 0.030 sec on http probe failed: 5, giveup 2024/11/19-15:16:56.645078 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00059/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.316 (submit 279.500 ms wait 0.030 s), input 4.762 KiB, send shutdown 2024/11/19-15:16:56.651334 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-01 2024/11/19-15:16:56.659840 139850978166528 [dcompact_etcd.cc:1773] job-00059/att-01: CleanFiles 0.008 sec : 0.000 2024/11/19-15:16:56.670030 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-01 2024/11/19-15:16:56.670113 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00054/att-01/compact.done) = No such file or directory: 9.012 sec on http probe failed: 3, retry 2024/11/19-15:16:56.699512 139833664079616 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4494: 231473 keys, 33325077 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.699620 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000616699566, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4494, "file_size": 33325077, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27025536, "index_size": 6298688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481244, "raw_average_key_size": 28, "raw_value_size": 27025522, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231473, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4494, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.721756 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-02 2024/11/19-15:16:56.721809 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00045/att-02/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:16:56.721825 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00045/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.825 sec, real 11.489 (submit 280.690 ms wait 11.202 s), input 126.825 MiB, send shutdown 2024/11/19-15:16:56.726664 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00045/att-02 2024/11/19-15:16:56.741456 139851148035840 [dcompact_etcd.cc:1773] job-00045/att-02: CleanFiles 0.015 sec : 0.000 2024/11/19-15:16:56.833965 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-02 2024/11/19-15:16:56.834053 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00046/att-02/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:16:56.834071 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00046/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.836 sec, real 11.517 (submit 304.680 ms wait 11.204 s), input 126.836 MiB, send shutdown 2024/11/19-15:16:56.841828 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00046/att-02 2024/11/19-15:16:56.951852 139850978166528 [dcompact_etcd.cc:1163] job-00059/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.292 sec, input = 4.762 KiB 2024/11/19-15:16:56.958501 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4495: 231476 keys, 33324949 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:56.958550 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000616958528, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4495, "file_size": 33324949, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27025856, "index_size": 6298240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481328, "raw_average_key_size": 28, "raw_value_size": 27025856, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231476, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4495, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:56.960194 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-02 2024/11/19-15:16:56.960249 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00059/att-02/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:16:56.967242 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-02 2024/11/19-15:16:56.967290 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00059/att-02/compact.done) = No such file or directory: 0.009 sec on http probe failed: 2, retry 2024/11/19-15:16:56.976253 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-02 2024/11/19-15:16:56.976297 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00059/att-02/compact.done) = No such file or directory: 0.016 sec on http probe failed: 3, retry 2024/11/19-15:16:56.981306 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-02 2024/11/19-15:16:56.981345 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00059/att-02/compact.done) = No such file or directory: 0.025 sec on http probe failed: 4, retry 2024/11/19-15:16:56.989600 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-02 2024/11/19-15:16:56.989660 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00059/att-02/compact.done) = No such file or directory: 0.030 sec on http probe failed: 5, giveup 2024/11/19-15:16:56.989676 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00059/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.330 (submit 291.862 ms wait 0.030 s), input 4.762 KiB, send shutdown 2024/11/19-15:16:56.998085 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00059/att-02 2024/11/19-15:16:57.022841 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-02 2024/11/19-15:16:57.022906 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00047/att-02/compact.done) = No such file or directory: 11.082 sec on http probe failed: 5, giveup 2024/11/19-15:16:57.022931 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00047/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 121.888 sec, real 11.367 (submit 280.504 ms wait 11.082 s), input 121.888 MiB, send shutdown 2024/11/19-15:16:57.031258 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00047/att-02 2024/11/19-15:16:57.061626 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00045) = success 2024/11/19-15:16:57.061636 139851148035840 [dcompact_etcd.cc:1773] job-00045/att-03: CleanFiles 0.320 sec : 0.000 2024/11/19-15:16:57.061660 139851148035840 (Original Log Time 2024/11/19-15:16:22.188999) EVENT_LOG_v1 {"time_micros": 1732000582188990, "job": 45, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4409, 4410, 4412, 4414], "score": 2.90411, "input_data_size": 134100224, "oldest_snapshot_seqno": 112255621} 2024/11/19-15:16:57.061667 139851148035840 [compaction_job.cc:2437] [default] [JOB 45] Compacting 4@1 files to L2, score 2.90, subcompactions 1 : 1 2024/11/19-15:16:57.061675 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 43 Base level 1, inputs: [4409(31MB) 4410(31MB) 4412(31MB) 4414(31MB)] 2024/11/19-15:16:57.061946 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004498.sst) /storage/mytopling/..//tmp/Topling-82038-Yc8Nsl 2024/11/19-15:16:57.116867 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004498.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:16:57.116976 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004498.sst): index pass time = 0.00's,7193.842'MB/sec index type = UintIndex_AllOne usrkeys = 64714 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:16:57.162211 139833664079616 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4496: 231451 keys, 33324949 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:57.162278 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000617162247, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4496, "file_size": 33324949, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27024000, "index_size": 6300096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480628, "raw_average_key_size": 28, "raw_value_size": 27023962, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231451, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000616, "oldest_key_time": 0, "file_creation_time": 1732000616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4496, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:57.167492 139851158521600 [dcompact_etcd.cc:1773] job-00046/att-02: CleanFiles 0.326 sec : 0.000 2024/11/19-15:16:57.167563 139850978166528 [dcompact_etcd.cc:1773] job-00059/att-02: CleanFiles 0.169 sec : 0.000 2024/11/19-15:16:57.167631 139851137550080 [dcompact_etcd.cc:1773] job-00047/att-02: CleanFiles 0.136 sec : 0.000 2024/11/19-15:16:57.167954 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4497: 231455 keys, 33325077 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:57.168007 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000617167978, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4497, "file_size": 33325077, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27027584, "index_size": 6296640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480740, "raw_average_key_size": 28, "raw_value_size": 27027562, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231455, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000617, "oldest_key_time": 0, "file_creation_time": 1732000617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4497, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:57.175717 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00059) = success 2024/11/19-15:16:57.175732 139850978166528 [dcompact_etcd.cc:1773] job-00059/att-03: CleanFiles 0.008 sec : 0.000 2024/11/19-15:16:57.175758 139850978166528 (Original Log Time 2024/11/19-15:16:55.998107) EVENT_LOG_v1 {"time_micros": 1732000615998098, "job": 59, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4081], "files_L3": [3816, 3952], "score": 0.314152, "input_data_size": 9211, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:57.175769 139850978166528 [compaction_job.cc:2437] [default] [JOB 59] Compacting 1@2 + 2@3 files to L3, score 0.31, subcompactions 1 : 1 2024/11/19-15:16:57.175781 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 57 Base level 2, inputs: [4081(3910B)], [3816(2234B) 3952(3067B)] 2024/11/19-15:16:57.175866 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00046) = success 2024/11/19-15:16:57.175882 139851158521600 [dcompact_etcd.cc:1773] job-00046/att-03: CleanFiles 0.008 sec : 0.000 2024/11/19-15:16:57.175908 139851158521600 (Original Log Time 2024/11/19-15:16:22.303395) EVENT_LOG_v1 {"time_micros": 1732000582303388, "job": 46, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4416, 4418, 4420, 4421], "score": 2.65433, "input_data_size": 134099392, "oldest_snapshot_seqno": 112255621} 2024/11/19-15:16:57.175918 139851158521600 [compaction_job.cc:2437] [default] [JOB 46] Compacting 4@1 files to L2, score 2.65, subcompactions 1 : 1 2024/11/19-15:16:57.175978 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 44 Base level 1, inputs: [4416(31MB) 4418(31MB) 4420(31MB) 4421(31MB)] 2024/11/19-15:16:57.176000 139850978166528 [compaction_job.cc:703] job-00059: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:16:57.176274 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00047) = success 2024/11/19-15:16:57.176286 139851137550080 [dcompact_etcd.cc:1773] job-00047/att-03: CleanFiles 0.009 sec : 0.000 2024/11/19-15:16:57.176308 139851137550080 (Original Log Time 2024/11/19-15:16:22.753012) EVENT_LOG_v1 {"time_micros": 1732000582753004, "job": 47, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4422, 4423, 4353, 4357], "score": 2.40455, "input_data_size": 128741824, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:57.176317 139851137550080 [compaction_job.cc:2437] [default] [JOB 47] Compacting 4@1 files to L2, score 2.40, subcompactions 1 : 1 2024/11/19-15:16:57.176328 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 45 Base level 1, inputs: [4422(31MB) 4423(26MB) 4353(31MB) 4357(31MB)] 2024/11/19-15:16:57.176544 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004502.sst) /storage/mytopling/..//tmp/Topling-82038-Rle70G 2024/11/19-15:16:57.177324 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004501.sst) /storage/mytopling/..//tmp/Topling-82038-ZOrc4i 2024/11/19-15:16:57.177538 139850978166528 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:16:57.177690 139850978166528 (Original Log Time 2024/11/19-15:16:57.175822) EVENT_LOG_v1 {"time_micros": 1732000617175795, "job": 59, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4081], "files_L3": [3816, 3952], "score": 0.314152, "input_data_size": 9211, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:57.177700 139850978166528 (Original Log Time 2024/11/19-15:16:57.177220) [compaction_job.cc:2022] [default] [JOB 59] Compacted 1@2 + 2@3 files to L3 => 0 bytes 2024/11/19-15:16:57.177706 139850978166528 (Original Log Time 2024/11/19-15:16:57.177504) [compaction_job.cc:1224] [default] compacted to: files[5 37 10 46 0 0 0] max score 0.31 (6 files need compaction), MB/sec: 49.5 rd, 0.0 wr, level 3, files in(1, 2) out(0 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(2.4) write-amplify(0.0) OK, records in: 37, records dropped: 37 output_compression: Snappy 2024/11/19-15:16:57.177727 139850978166528 (Original Log Time 2024/11/19-15:16:57.177530) EVENT_LOG_v1 {"time_micros": 1732000617177515, "cf": "default", "job": 59, "event": "compaction_finished", "compaction_time_micros": 186, "compaction_time_cpu_micros": 159, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 37, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 37, 10, 46, 0, 0, 0]} 2024/11/19-15:16:57.177911 139850978166528 [compaction_job.cc:2437] [default] [JOB 61] Compacting 1@2 files to L3, score 0.31, subcompactions 1 : 1 2024/11/19-15:16:57.177938 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 59 Base level 2, inputs: [4462(61MB)] 2024/11/19-15:16:57.188845 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004498.sst): first pass time = 0.13's, 528.895'MB/sec 2024/11/19-15:16:57.188909 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004498.sst): old prealloc_size = 73819750, real_size = 66042344 2024/11/19-15:16:57.188943 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004498.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64714 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.02 avg-zval =1020.27 2024/11/19-15:16:57.260698 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004501.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:16:57.261652 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004502.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:16:57.265912 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004501.sst): index pass time = 0.01's, 100.143'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64714 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:16:57.266864 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004502.sst): index pass time = 0.01's, 99.909'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64715 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:16:57.268417 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004498.sst): second pass time = 0.00's,169191017.995'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.08's, 830.677'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 7.317'MB/sec (index lex order gen) rebuild zvType time = 0.06's, 0.280'MB/sec write SST data time = 0.02's, 3057.212'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 169191017.995'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64714 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.02 avg-zval = 1020.27 usrkeys = 64714 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.02 avg-zval = 1020.27 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.6250 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 2.135462 GB avg = 0.395 KB (by entry num) total key len = 0.135316 GB avg = 0.025 KB total ukey len = 0.092103 GB avg = 0.017 KB total ukey num = 0.005401635 Billion total entry num = 0.005401635 Billion write speed all = 18.531251903 MB/sec (with seq num) write speed all = 18.178600752 MB/sec (without seq num) 2024/11/19-15:16:57.274419 139851148035840 [compaction_job.cc:1948] [default] [JOB 45] Generated table #4498: 64714 keys, 66027123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:57.274493 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000617274455, "cf_name": "default", "job": 45, "event": "table_file_creation", "file_number": 4498, "file_size": 66027123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025840, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294280, "raw_average_key_size": 20, "raw_value_size": 65815306, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64714, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000617, "oldest_key_time": 0, "file_creation_time": 1732000617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4498, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:57.283004 139833664079616 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4499: 231375 keys, 33325013 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:57.283052 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000617283030, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4499, "file_size": 33325013, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27028288, "index_size": 6295872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6478500, "raw_average_key_size": 28, "raw_value_size": 27028278, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231375, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000617, "oldest_key_time": 0, "file_creation_time": 1732000617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4499, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:57.301007 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004504.sst) /storage/mytopling/..//tmp/Topling-82038-B9e1dw 2024/11/19-15:16:57.353169 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004501.sst): first pass time = 0.18's, 381.684'MB/sec 2024/11/19-15:16:57.355048 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004501.sst): old prealloc_size = 73819750, real_size = 66151712 2024/11/19-15:16:57.355067 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004502.sst): first pass time = 0.18's, 375.947'MB/sec 2024/11/19-15:16:57.355109 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004501.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64714 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:16:57.355122 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004502.sst): old prealloc_size = 73819750, real_size = 66151888 2024/11/19-15:16:57.355157 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004501.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.094 sec, Key+Value = 0.067 GB 2024/11/19-15:16:57.355186 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004502.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64715 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.99 avg-zval =1020.25 2024/11/19-15:16:57.355244 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004502.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.094 sec, Key+Value = 0.067 GB 2024/11/19-15:16:57.375496 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4500: 231258 keys, 33325205 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:57.375577 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000617375543, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4500, "file_size": 33325205, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27033536, "index_size": 6290816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6475224, "raw_average_key_size": 28, "raw_value_size": 27033524, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231258, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000617, "oldest_key_time": 0, "file_creation_time": 1732000617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4500, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:57.386437 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004504.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:16:57.393938 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004504.sst): index pass time = 0.01's, 68.178'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63527 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:16:57.413289 139833664079616 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4503: 231310 keys, 33325141 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:57.413348 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000617413325, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4503, "file_size": 33325141, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27030144, "index_size": 6294144, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6476680, "raw_average_key_size": 28, "raw_value_size": 27030092, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231310, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000617, "oldest_key_time": 0, "file_creation_time": 1732000617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4503, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:57.431834 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004501.sst): second pass time = 0.00's,134315561.224'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.08's, 841.678'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2064.439'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.297'MB/sec write SST data time = 0.02's, 2747.723'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 134315561.224'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64714 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 64714 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0464 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 2.201277 GB avg = 0.403 KB (by entry num) total key len = 0.136610 GB avg = 0.025 KB total ukey len = 0.092879 GB avg = 0.017 KB total ukey num = 0.005466349 Billion total entry num = 0.005466349 Billion write speed all = 19.053501298 MB/sec (with seq num) write speed all = 18.697100522 MB/sec (without seq num) 2024/11/19-15:16:57.444662 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004502.sst): second pass time = 0.00's,160523917.073'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 738.875'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1770.523'MB/sec (index lex order gen) rebuild zvType time = 0.07's, 0.249'MB/sec write SST data time = 0.02's, 2715.098'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 160523917.073'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64715 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.25 usrkeys = 64715 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8063 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0465 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 2.267091 GB avg = 0.410 KB (by entry num) total key len = 0.137905 GB avg = 0.025 KB total ukey len = 0.093656 GB avg = 0.017 KB total ukey num = 0.005531064 Billion total entry num = 0.005531064 Billion write speed all = 19.598383589 MB/sec (with seq num) write speed all = 19.237801155 MB/sec (without seq num) 2024/11/19-15:16:57.450931 139851158521600 [compaction_job.cc:1948] [default] [JOB 46] Generated table #4501: 64714 keys, 66136941 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:57.451003 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000617450968, "cf_name": "default", "job": 46, "event": "table_file_creation", "file_number": 4501, "file_size": 66136941, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025168, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294280, "raw_average_key_size": 20, "raw_value_size": 65814625, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64714, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000617, "oldest_key_time": 0, "file_creation_time": 1732000617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4501, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:57.458152 139850978166528 [dcompact_etcd.cc:1163] job-00061/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 62.907 MiB 2024/11/19-15:16:57.463068 139851137550080 [compaction_job.cc:1948] [default] [JOB 47] Generated table #4502: 64715 keys, 66137069 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:57.463137 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000617463102, "cf_name": "default", "job": 47, "event": "table_file_creation", "file_number": 4502, "file_size": 66137069, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025344, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294300, "raw_average_key_size": 20, "raw_value_size": 65814806, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64715, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000617, "oldest_key_time": 0, "file_creation_time": 1732000617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4502, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:57.472746 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004504.sst): first pass time = 0.17's, 383.638'MB/sec 2024/11/19-15:16:57.472799 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004504.sst): old prealloc_size = 73819750, real_size = 64936720 2024/11/19-15:16:57.472874 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004504.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63527 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0646 GB zip-val = 0.0648 GB avg-val =1016.98 avg-zval =1020.23 2024/11/19-15:16:57.472938 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004504.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.087 sec, Key+Value = 0.066 GB 2024/11/19-15:16:57.475545 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004507.sst) /storage/mytopling/..//tmp/Topling-82038-iCeFwh 2024/11/19-15:16:57.487814 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004508.sst) /storage/mytopling/..//tmp/Topling-82038-4TElSD 2024/11/19-15:16:57.491241 139833664079616 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4506: 131035 keys, 18910549 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:57.491292 139833664079616 EVENT_LOG_v1 {"time_micros": 1732000617491269, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4506, "file_size": 18910549, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 15283648, "index_size": 3626048, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3668980, "raw_average_key_size": 28, "raw_value_size": 15283634, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 131035, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000617, "oldest_key_time": 0, "file_creation_time": 1732000617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4506, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:57.538140 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004507.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:16:57.538252 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004507.sst): index pass time = 0.00's,6394.505'MB/sec index type = UintIndex_AllOne usrkeys = 63539 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:16:57.543393 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4505: 231470 keys, 33324949 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:57.543448 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000617543426, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4505, "file_size": 33324949, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27026368, "index_size": 6297728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481160, "raw_average_key_size": 28, "raw_value_size": 27026328, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231470, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000617, "oldest_key_time": 0, "file_creation_time": 1732000617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4505, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:57.556011 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004504.sst): second pass time = 0.00's,223549024.221'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.08's, 780.443'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1457.796'MB/sec (index lex order gen) rebuild zvType time = 0.06's, 0.265'MB/sec write SST data time = 0.02's, 2807.696'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 223549024.221'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63527 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 1020.23 usrkeys = 63527 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 1020.23 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7915 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0648 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index = 7.0398 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 2.331697 GB avg = 0.417 KB (by entry num) total key len = 0.139175 GB avg = 0.025 KB total ukey len = 0.094418 GB avg = 0.017 KB total ukey num = 0.005594591 Billion total entry num = 0.005594591 Billion write speed all = 20.116956485 MB/sec (with seq num) write speed all = 19.752563237 MB/sec (without seq num) 2024/11/19-15:16:57.562200 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004508.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.061 GB 2024/11/19-15:16:57.562309 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004508.sst): index pass time = 0.00's,5790.000'MB/sec index type = UintIndex_AllOne usrkeys = 58534 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:16:57.605603 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004507.sst): first pass time = 0.13's, 506.690'MB/sec 2024/11/19-15:16:57.605657 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004507.sst): old prealloc_size = 73819750, real_size = 64839960 2024/11/19-15:16:57.605679 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004507.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63539 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0646 GB zip-val = 0.0648 GB avg-val =1016.97 avg-zval =1020.22 2024/11/19-15:16:57.625003 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004508.sst): first pass time = 0.14's, 442.535'MB/sec 2024/11/19-15:16:57.625054 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004508.sst): old prealloc_size = 73819750, real_size = 59734920 2024/11/19-15:16:57.625077 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004508.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 58534 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0595 GB zip-val = 0.0597 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:16:57.646756 139851148035840 [compaction_job.cc:1948] [default] [JOB 45] Generated table #4504: 63527 keys, 64922157 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:57.646841 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000617646806, "cf_name": "default", "job": 45, "event": "table_file_creation", "file_number": 4504, "file_size": 64922157, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64812352, "index_size": 108288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1270540, "raw_average_key_size": 20, "raw_value_size": 64605668, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63527, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000617, "oldest_key_time": 0, "file_creation_time": 1732000617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4504, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:57.668504 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004507.sst): second pass time = 0.00's,140777775.599'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1032.058'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.050'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.340'MB/sec write SST data time = 0.02's, 4020.009'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 140777775.599'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63539 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.97 avg-zval = 1020.22 usrkeys = 63539 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.97 avg-zval = 1020.22 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0000 GB value = 0.0648 GB dict = 0.00 MB all = 0.0648 GB } UnZip/Zip{ index =3971.1875 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 2.396314 GB avg = 0.424 KB (by entry num) total key len = 0.140446 GB avg = 0.025 KB total ukey len = 0.095181 GB avg = 0.017 KB total ukey num = 0.005658130 Billion total entry num = 0.005658130 Billion write speed all = 20.634493455 MB/sec (with seq num) write speed all = 20.266298917 MB/sec (without seq num) 2024/11/19-15:16:57.670167 139851148035840 [compaction_job.cc:703] job-00045: subcompact[0], size: 0.130949 G, files: /storage/mytopling/.rocksdb [004498,004504] 2024/11/19-15:16:57.673125 139851148035840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:16:57.673302 139851148035840 (Original Log Time 2024/11/19-15:16:57.061711) EVENT_LOG_v1 {"time_micros": 1732000617061688, "job": 45, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4409, 4410, 4412, 4414], "score": 2.90411, "input_data_size": 134100224, "oldest_snapshot_seqno": 112255621} 2024/11/19-15:16:57.673310 139851148035840 (Original Log Time 2024/11/19-15:16:57.672789) [compaction_job.cc:2022] [default] [JOB 45] Compacted 4@1 files to L2 => 130949280 bytes 2024/11/19-15:16:57.673315 139851148035840 (Original Log Time 2024/11/19-15:16:57.673079) [compaction_job.cc:1224] [default] compacted to: files[5 33 12 46 0 0 0] max score 0.31 (7 files need compaction), MB/sec: 220.4 rd, 215.2 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(124.9 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128241, records dropped: 0 output_compression: Snappy 2024/11/19-15:16:57.673338 139851148035840 (Original Log Time 2024/11/19-15:16:57.673116) EVENT_LOG_v1 {"time_micros": 1732000617673097, "cf": "default", "job": 45, "event": "compaction_finished", "compaction_time_micros": 608473, "compaction_time_cpu_micros": 518153, "output_level": 2, "num_output_files": 2, "total_output_size": 130949280, "num_input_records": 128241, "num_output_records": 128241, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 33, 12, 46, 0, 0, 0]} 2024/11/19-15:16:57.684509 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004508.sst): second pass time = 0.00's,106683695.341'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1004.681'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.920'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.364'MB/sec write SST data time = 0.02's, 3104.472'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.060'GB zip my value throughput = 106683695.341'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 58534 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 58534 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0007 GB value = 0.0595 GB dict = 0.00 MB all = 0.0607 GB } __ZipSize{ index = 0.0000 GB value = 0.0597 GB dict = 0.00 MB all = 0.0597 GB } UnZip/Zip{ index =3658.3750 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 2.455844 GB avg = 0.430 KB (by entry num) total key len = 0.141617 GB avg = 0.025 KB total ukey len = 0.095883 GB avg = 0.017 KB total ukey num = 0.005716664 Billion total entry num = 0.005716664 Billion write speed all = 21.125485660 MB/sec (with seq num) write speed all = 20.753530610 MB/sec (without seq num) 2024/11/19-15:16:57.879177 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4509: 231441 keys, 33324949 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:57.879245 139851158521600 [compaction_job.cc:1948] [default] [JOB 46] Generated table #4507: 63539 keys, 64825011 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:57.879262 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000617879227, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4509, "file_size": 33324949, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27022848, "index_size": 6301248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480348, "raw_average_key_size": 28, "raw_value_size": 27022806, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231441, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000617, "oldest_key_time": 0, "file_creation_time": 1732000617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4509, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:57.879298 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000617879276, "cf_name": "default", "job": 46, "event": "table_file_creation", "file_number": 4507, "file_size": 64825011, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64823712, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1270780, "raw_average_key_size": 20, "raw_value_size": 64616999, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63539, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000617, "oldest_key_time": 0, "file_creation_time": 1732000617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4507, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:57.879505 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004307.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.879531 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000617879528, "job": 45, "event": "table_file_deletion", "file_number": 4307} 2024/11/19-15:16:57.879613 139851137550080 [compaction_job.cc:1948] [default] [JOB 47] Generated table #4508: 58534 keys, 59721267 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:57.879660 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000617879633, "cf_name": "default", "job": 47, "event": "table_file_creation", "file_number": 4508, "file_size": 59721267, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 59719952, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1170680, "raw_average_key_size": 20, "raw_value_size": 59529502, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 58534, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000617, "oldest_key_time": 0, "file_creation_time": 1732000617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4508, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:57.889628 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004308.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.889657 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000617889654, "job": 45, "event": "table_file_deletion", "file_number": 4308} 2024/11/19-15:16:57.899702 139851158521600 [compaction_job.cc:703] job-00046: subcompact[0], size: 0.130962 G, files: /storage/mytopling/.rocksdb [004501,004507] 2024/11/19-15:16:57.899799 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004311.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.899826 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000617899823, "job": 45, "event": "table_file_deletion", "file_number": 4311} 2024/11/19-15:16:57.900416 139851158521600 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:16:57.900563 139851158521600 (Original Log Time 2024/11/19-15:16:57.176020) EVENT_LOG_v1 {"time_micros": 1732000617175991, "job": 46, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4416, 4418, 4420, 4421], "score": 2.65433, "input_data_size": 134099392, "oldest_snapshot_seqno": 112255621} 2024/11/19-15:16:57.900570 139851158521600 (Original Log Time 2024/11/19-15:16:57.900180) [compaction_job.cc:2022] [default] [JOB 46] Compacted 4@1 files to L2 => 130961952 bytes 2024/11/19-15:16:57.900573 139851158521600 (Original Log Time 2024/11/19-15:16:57.900389) [compaction_job.cc:1224] [default] compacted to: files[5 29 14 46 0 0 0] max score 0.31 (9 files need compaction), MB/sec: 185.3 rd, 181.0 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(124.9 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128253, records dropped: 0 output_compression: Snappy 2024/11/19-15:16:57.900592 139851158521600 (Original Log Time 2024/11/19-15:16:57.900409) EVENT_LOG_v1 {"time_micros": 1732000617900398, "cf": "default", "job": 46, "event": "compaction_finished", "compaction_time_micros": 723695, "compaction_time_cpu_micros": 497703, "output_level": 2, "num_output_files": 2, "total_output_size": 130961952, "num_input_records": 128253, "num_output_records": 128253, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 29, 14, 46, 0, 0, 0]} 2024/11/19-15:16:57.901410 139851137550080 [compaction_job.cc:703] job-00047: subcompact[0], size: 0.125858 G, files: /storage/mytopling/.rocksdb [004502,004508] 2024/11/19-15:16:57.902127 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:16:57.902280 139851137550080 (Original Log Time 2024/11/19-15:16:57.176359) EVENT_LOG_v1 {"time_micros": 1732000617176337, "job": 47, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4422, 4423, 4353, 4357], "score": 2.40455, "input_data_size": 128741824, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:57.902289 139851137550080 (Original Log Time 2024/11/19-15:16:57.901860) [compaction_job.cc:2022] [default] [JOB 47] Compacted 4@1 files to L2 => 125858336 bytes 2024/11/19-15:16:57.902294 139851137550080 (Original Log Time 2024/11/19-15:16:57.902095) [compaction_job.cc:1224] [default] compacted to: files[5 25 16 46 0 0 0] max score 0.31 (11 files need compaction), MB/sec: 177.6 rd, 173.6 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(122.8, 0.0 +0.0 blob) out(120.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 123249, records dropped: 0 output_compression: Snappy 2024/11/19-15:16:57.902318 139851137550080 (Original Log Time 2024/11/19-15:16:57.902121) EVENT_LOG_v1 {"time_micros": 1732000617902107, "cf": "default", "job": 47, "event": "compaction_finished", "compaction_time_micros": 725065, "compaction_time_cpu_micros": 512927, "output_level": 2, "num_output_files": 2, "total_output_size": 125858336, "num_input_records": 123249, "num_output_records": 123249, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 25, 16, 46, 0, 0, 0]} 2024/11/19-15:16:57.908147 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004356.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.908173 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000617908171, "job": 47, "event": "table_file_deletion", "file_number": 4356} 2024/11/19-15:16:57.908552 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004315.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.908576 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000617908574, "job": 46, "event": "table_file_deletion", "file_number": 4315} 2024/11/19-15:16:57.911023 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004314.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.911048 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000617911045, "job": 45, "event": "table_file_deletion", "file_number": 4314} 2024/11/19-15:16:57.911089 139834427442944 (Original Log Time 2024/11/19-15:16:48.707663) [db_impl_compaction_flush.cc:2043] [default] Manual compaction starting 2024/11/19-15:16:57.911228 139851148035840 [compaction_job.cc:2437] [default] [JOB 62] Compacting 1@1 files to L2, score 0.31, subcompactions 1 : 1 2024/11/19-15:16:57.911239 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 62 Base level 1, inputs: [4445(3396B)] 2024/11/19-15:16:57.913722 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004363.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.913744 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000617913742, "job": 47, "event": "table_file_deletion", "file_number": 4363} 2024/11/19-15:16:57.915888 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004322.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.915910 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000617915908, "job": 46, "event": "table_file_deletion", "file_number": 4322} 2024/11/19-15:16:57.918072 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004370.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.918092 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000617918090, "job": 47, "event": "table_file_deletion", "file_number": 4370} 2024/11/19-15:16:57.922228 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004331.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.922249 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000617922247, "job": 46, "event": "table_file_deletion", "file_number": 4331} 2024/11/19-15:16:57.922481 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004373.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.922501 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000617922499, "job": 47, "event": "table_file_deletion", "file_number": 4373} 2024/11/19-15:16:57.926568 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004378.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.926587 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000617926585, "job": 47, "event": "table_file_deletion", "file_number": 4378} 2024/11/19-15:16:57.928690 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004338.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.928710 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000617928708, "job": 46, "event": "table_file_deletion", "file_number": 4338} 2024/11/19-15:16:57.928828 139851158521600 [compaction_job.cc:2437] [default] [JOB 63] Compacting 1@1 files to L2, score 0.31, subcompactions 1 : 1 2024/11/19-15:16:57.928844 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 62 Base level 1, inputs: [4448(3588B)] 2024/11/19-15:16:57.930632 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004382.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.930651 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000617930649, "job": 47, "event": "table_file_deletion", "file_number": 4382} 2024/11/19-15:16:57.934597 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004387.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.934616 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000617934614, "job": 47, "event": "table_file_deletion", "file_number": 4387} 2024/11/19-15:16:57.937229 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004391.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.937247 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000617937245, "job": 47, "event": "table_file_deletion", "file_number": 4391} 2024/11/19-15:16:57.937384 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004424.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:16:57.937393 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000617937392, "job": 47, "event": "table_file_deletion", "file_number": 4424} 2024/11/19-15:16:57.937501 139851137550080 [compaction_job.cc:2437] [default] [JOB 64] Compacting 1@2 + 2@3 files to L3, score 0.31, subcompactions 1 : 1 2024/11/19-15:16:57.937510 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 62 Base level 2, inputs: [4082(30KB)], [3802(17KB) 3953(22KB)] 2024/11/19-15:16:57.992557 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4510: 231341 keys, 33325141 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:57.992602 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000617992581, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4510, "file_size": 33325141, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27029824, "index_size": 6294464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6477548, "raw_average_key_size": 28, "raw_value_size": 27029766, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231341, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000617, "oldest_key_time": 0, "file_creation_time": 1732000617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4510, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.094569 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4511: 231358 keys, 33325013 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.094622 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618094600, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4511, "file_size": 33325013, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27031104, "index_size": 6293056, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6478024, "raw_average_key_size": 28, "raw_value_size": 27031080, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231358, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000617, "oldest_key_time": 0, "file_creation_time": 1732000617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4511, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.109952 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-02 2024/11/19-15:16:58.110041 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00051/att-02/compact.done) = No such file or directory: 6.881 sec on http probe failed: 3, retry 2024/11/19-15:16:58.195650 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4512: 231436 keys, 33325013 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.195701 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618195680, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4512, "file_size": 33325013, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27024128, "index_size": 6300032, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480208, "raw_average_key_size": 28, "raw_value_size": 27024116, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231436, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4512, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.197666 139851148035840 [dcompact_etcd.cc:1163] job-00062/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.286 sec, input = 236 B 2024/11/19-15:16:58.214654 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-00 2024/11/19-15:16:58.214711 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00062/att-00/compact.done) = No such file or directory: 0.007 sec on http probe failed: 1, retry 2024/11/19-15:16:58.224623 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-00 2024/11/19-15:16:58.224675 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00062/att-00/compact.done) = No such file or directory: 0.017 sec on http probe failed: 2, retry 2024/11/19-15:16:58.236355 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-00 2024/11/19-15:16:58.236402 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00062/att-00/compact.done) = No such file or directory: 0.027 sec on http probe failed: 3, retry 2024/11/19-15:16:58.241686 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-00 2024/11/19-15:16:58.241733 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00062/att-00/compact.done) = No such file or directory: 0.039 sec on http probe failed: 4, retry 2024/11/19-15:16:58.244426 139851158521600 [dcompact_etcd.cc:1163] job-00063/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.315 sec, input = 525 B 2024/11/19-15:16:58.252529 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-00 2024/11/19-15:16:58.252574 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00062/att-00/compact.done) = No such file or directory: 0.044 sec on http probe failed: 5, giveup 2024/11/19-15:16:58.252587 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00062/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.341 (submit 285.867 ms wait 0.044 s), input 236 B, send shutdown 2024/11/19-15:16:58.253409 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-00 2024/11/19-15:16:58.253472 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00063/att-00/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:16:58.258509 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-00 2024/11/19-15:16:58.258555 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00063/att-00/compact.done) = No such file or directory: 0.009 sec on http probe failed: 2, retry 2024/11/19-15:16:58.260888 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-00 2024/11/19-15:16:58.261092 139851148035840 [dcompact_etcd.cc:1773] job-00062/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:58.261524 139851137550080 [dcompact_etcd.cc:1163] job-00064/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.324 sec, input = 50.001 KiB 2024/11/19-15:16:58.267094 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-00 2024/11/19-15:16:58.267142 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00063/att-00/compact.done) = No such file or directory: 0.014 sec on http probe failed: 3, retry 2024/11/19-15:16:58.271280 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-00 2024/11/19-15:16:58.271334 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00064/att-00/compact.done) = No such file or directory: 0.001 sec on http probe failed: 1, retry 2024/11/19-15:16:58.272165 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-00 2024/11/19-15:16:58.272211 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00063/att-00/compact.done) = No such file or directory: 0.023 sec on http probe failed: 4, retry 2024/11/19-15:16:58.279950 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-00 2024/11/19-15:16:58.279993 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00063/att-00/compact.done) = No such file or directory: 0.028 sec on http probe failed: 5, giveup 2024/11/19-15:16:58.280006 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00063/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.351 (submit 315.137 ms wait 0.028 s), input 525 B, send shutdown 2024/11/19-15:16:58.281227 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-00 2024/11/19-15:16:58.281275 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00064/att-00/compact.done) = No such file or directory: 0.011 sec on http probe failed: 2, retry 2024/11/19-15:16:58.285511 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-00 2024/11/19-15:16:58.285686 139851158521600 [dcompact_etcd.cc:1773] job-00063/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:58.291268 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-00 2024/11/19-15:16:58.291313 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00064/att-00/compact.done) = No such file or directory: 0.021 sec on http probe failed: 3, retry 2024/11/19-15:16:58.296685 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4513: 231469 keys, 33325013 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.296730 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618296709, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4513, "file_size": 33325013, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27027584, "index_size": 6296576, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481132, "raw_average_key_size": 28, "raw_value_size": 27027550, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231469, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4513, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.298026 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-00 2024/11/19-15:16:58.298069 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00064/att-00/compact.done) = No such file or directory: 0.031 sec on http probe failed: 4, retry 2024/11/19-15:16:58.307365 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-00 2024/11/19-15:16:58.307416 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00064/att-00/compact.done) = No such file or directory: 0.038 sec on http probe failed: 5, giveup 2024/11/19-15:16:58.307431 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00064/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.370 (submit 323.600 ms wait 0.038 s), input 50.001 KiB, send shutdown 2024/11/19-15:16:58.311586 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-00 2024/11/19-15:16:58.311767 139851137550080 [dcompact_etcd.cc:1773] job-00064/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:16:58.398053 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4514: 231244 keys, 33325205 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.398107 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618398085, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4514, "file_size": 33325205, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27033536, "index_size": 6290816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6474832, "raw_average_key_size": 28, "raw_value_size": 27033512, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231244, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4514, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.415730 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4515: 39594 keys, 5793490 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.415776 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618415756, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4515, "file_size": 5793490, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 4600320, "index_size": 1192320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1108632, "raw_average_key_size": 28, "raw_value_size": 4600288, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 39594, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4515, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.449149 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4516: 100000 keys, 9267540 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.449194 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618449174, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4516, "file_size": 9267540, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 6701312, "index_size": 2565376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1600000, "raw_average_key_size": 16, "raw_value_size": 6701254, "raw_average_value_size": 67, "num_data_blocks": 1, "num_entries": 100000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4516, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.480552 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4517: 32060 keys, 33525202 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.480598 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618480577, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4517, "file_size": 33525202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32605312, "index_size": 919040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32605259, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4517, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.511875 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4518: 32107 keys, 33525138 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.511929 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618511900, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4518, "file_size": 33525138, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32654144, "index_size": 870144, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642140, "raw_average_key_size": 20, "raw_value_size": 32654096, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32107, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4518, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.531780 139851148035840 [dcompact_etcd.cc:1163] job-00062/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.271 sec, input = 236 B 2024/11/19-15:16:58.540036 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-01 2024/11/19-15:16:58.540093 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00062/att-01/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:16:58.543205 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4519: 32057 keys, 33524690 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.543247 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618543227, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4519, "file_size": 33524690, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32602816, "index_size": 921024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602805, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4519, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.551292 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-01 2024/11/19-15:16:58.551345 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00062/att-01/compact.done) = No such file or directory: 0.008 sec on http probe failed: 2, retry 2024/11/19-15:16:58.555638 139851158521600 [dcompact_etcd.cc:1163] job-00063/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.270 sec, input = 525 B 2024/11/19-15:16:58.559392 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-01 2024/11/19-15:16:58.559440 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00062/att-01/compact.done) = No such file or directory: 0.020 sec on http probe failed: 3, retry 2024/11/19-15:16:58.564980 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-01 2024/11/19-15:16:58.565024 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00062/att-01/compact.done) = No such file or directory: 0.028 sec on http probe failed: 4, retry 2024/11/19-15:16:58.565378 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-01 2024/11/19-15:16:58.565449 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00063/att-01/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:16:58.572520 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-01 2024/11/19-15:16:58.572566 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00062/att-01/compact.done) = No such file or directory: 0.033 sec on http probe failed: 5, giveup 2024/11/19-15:16:58.572580 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00062/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.311 (submit 270.617 ms wait 0.033 s), input 236 B, send shutdown 2024/11/19-15:16:58.573308 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-01 2024/11/19-15:16:58.573369 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00063/att-01/compact.done) = No such file or directory: 0.010 sec on http probe failed: 2, retry 2024/11/19-15:16:58.574768 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4520: 32057 keys, 33524690 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.574810 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618574789, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4520, "file_size": 33524690, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32603200, "index_size": 920640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32603167, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4520, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.588861 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-01 2024/11/19-15:16:58.588972 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-01 2024/11/19-15:16:58.589027 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00063/att-01/compact.done) = No such file or directory: 0.018 sec on http probe failed: 3, retry 2024/11/19-15:16:58.596297 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-01 2024/11/19-15:16:58.596352 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00063/att-01/compact.done) = No such file or directory: 0.033 sec on http probe failed: 4, retry 2024/11/19-15:16:58.597670 139851137550080 [dcompact_etcd.cc:1163] job-00064/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.286 sec, input = 50.001 KiB 2024/11/19-15:16:58.603894 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-01 2024/11/19-15:16:58.603949 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00063/att-01/compact.done) = No such file or directory: 0.041 sec on http probe failed: 5, giveup 2024/11/19-15:16:58.603963 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00063/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.318 (submit 269.880 ms wait 0.041 s), input 525 B, send shutdown 2024/11/19-15:16:58.605968 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-01 2024/11/19-15:16:58.606030 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00064/att-01/compact.done) = No such file or directory: 0.001 sec on http probe failed: 1, retry 2024/11/19-15:16:58.611807 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-01 2024/11/19-15:16:58.612302 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-01 2024/11/19-15:16:58.612349 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00064/att-01/compact.done) = No such file or directory: 0.009 sec on http probe failed: 2, retry 2024/11/19-15:16:58.617995 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-01 2024/11/19-15:16:58.618037 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00064/att-01/compact.done) = No such file or directory: 0.016 sec on http probe failed: 3, retry 2024/11/19-15:16:58.623750 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-01 2024/11/19-15:16:58.623792 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00064/att-01/compact.done) = No such file or directory: 0.021 sec on http probe failed: 4, retry 2024/11/19-15:16:58.629573 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-01 2024/11/19-15:16:58.629614 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00064/att-01/compact.done) = No such file or directory: 0.027 sec on http probe failed: 5, giveup 2024/11/19-15:16:58.629626 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00064/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.318 (submit 285.819 ms wait 0.027 s), input 50.001 KiB, send shutdown 2024/11/19-15:16:58.633584 139851148035840 [dcompact_etcd.cc:1773] job-00062/att-01: CleanFiles 0.045 sec : 0.000 2024/11/19-15:16:58.637378 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-01 2024/11/19-15:16:58.642842 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4521: 32151 keys, 33525138 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.642895 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618642866, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4521, "file_size": 33525138, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32696704, "index_size": 827584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32696694, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4521, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.642908 139851158521600 [dcompact_etcd.cc:1773] job-00063/att-01: CleanFiles 0.031 sec : 0.000 2024/11/19-15:16:58.653120 139851137550080 [dcompact_etcd.cc:1773] job-00064/att-01: CleanFiles 0.016 sec : 0.000 2024/11/19-15:16:58.677754 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-01 2024/11/19-15:16:58.677827 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00054/att-01/compact.done) = No such file or directory: 11.017 sec on http probe failed: 4, retry 2024/11/19-15:16:58.693198 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4522: 32057 keys, 33524626 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.693245 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618693224, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4522, "file_size": 33524626, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32602048, "index_size": 921728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601993, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4522, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.725909 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4523: 32060 keys, 33525202 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.725972 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618725951, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4523, "file_size": 33525202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32603968, "index_size": 920384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32603958, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4523, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.771336 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4524: 32152 keys, 33525074 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.771388 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618771367, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4524, "file_size": 33525074, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32697664, "index_size": 826560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643040, "raw_average_key_size": 20, "raw_value_size": 32697630, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32152, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4524, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.841192 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4525: 32058 keys, 33524690 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.841247 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618841225, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4525, "file_size": 33524690, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32602496, "index_size": 921344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602481, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4525, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.879454 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4526: 32059 keys, 33524946 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.879505 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618879484, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4526, "file_size": 33524946, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32603072, "index_size": 921024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603014, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4526, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.910918 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4527: 32124 keys, 33524434 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:58.910978 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000618910957, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4527, "file_size": 33524434, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32670528, "index_size": 853056, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642480, "raw_average_key_size": 20, "raw_value_size": 32670465, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32124, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4527, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:58.933967 139851137550080 [dcompact_etcd.cc:1163] job-00064/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 50.001 KiB 2024/11/19-15:16:58.935392 139851148035840 [dcompact_etcd.cc:1163] job-00062/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.302 sec, input = 236 B 2024/11/19-15:16:58.939157 139851158521600 [dcompact_etcd.cc:1163] job-00063/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.296 sec, input = 525 B 2024/11/19-15:16:58.940250 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-02 2024/11/19-15:16:58.940310 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00064/att-02/compact.done) = No such file or directory: 0.001 sec on http probe failed: 1, retry 2024/11/19-15:16:58.943839 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-02 2024/11/19-15:16:58.943894 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00062/att-02/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:16:58.949804 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-02 2024/11/19-15:16:58.949869 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00064/att-02/compact.done) = No such file or directory: 0.007 sec on http probe failed: 2, retry 2024/11/19-15:16:58.952332 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-02 2024/11/19-15:16:58.952380 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00062/att-02/compact.done) = No such file or directory: 0.009 sec on http probe failed: 2, retry 2024/11/19-15:16:58.952428 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-02 2024/11/19-15:16:58.952481 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00063/att-02/compact.done) = No such file or directory: 0.005 sec on http probe failed: 1, retry 2024/11/19-15:16:58.959369 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-02 2024/11/19-15:16:58.959416 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00064/att-02/compact.done) = No such file or directory: 0.017 sec on http probe failed: 3, retry 2024/11/19-15:16:58.959866 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-02 2024/11/19-15:16:58.959905 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00063/att-02/compact.done) = No such file or directory: 0.013 sec on http probe failed: 2, retry 2024/11/19-15:16:58.960772 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-02 2024/11/19-15:16:58.960817 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00062/att-02/compact.done) = No such file or directory: 0.017 sec on http probe failed: 3, retry 2024/11/19-15:16:58.967231 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-02 2024/11/19-15:16:58.967280 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00063/att-02/compact.done) = No such file or directory: 0.021 sec on http probe failed: 3, retry 2024/11/19-15:16:58.967461 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-02 2024/11/19-15:16:58.967513 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00064/att-02/compact.done) = No such file or directory: 0.027 sec on http probe failed: 4, retry 2024/11/19-15:16:58.969148 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-02 2024/11/19-15:16:58.969190 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00062/att-02/compact.done) = No such file or directory: 0.026 sec on http probe failed: 4, retry 2024/11/19-15:16:58.976134 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-02 2024/11/19-15:16:58.976184 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00063/att-02/compact.done) = No such file or directory: 0.028 sec on http probe failed: 4, retry 2024/11/19-15:16:58.976578 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-02 2024/11/19-15:16:58.976629 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00064/att-02/compact.done) = No such file or directory: 0.035 sec on http probe failed: 5, giveup 2024/11/19-15:16:58.976644 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00064/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.323 (submit 280.755 ms wait 0.035 s), input 50.001 KiB, send shutdown 2024/11/19-15:16:58.977005 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-02 2024/11/19-15:16:58.977059 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00062/att-02/compact.done) = No such file or directory: 0.034 sec on http probe failed: 5, giveup 2024/11/19-15:16:58.977076 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00062/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.343 (submit 301.711 ms wait 0.034 s), input 236 B, send shutdown 2024/11/19-15:16:58.980553 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-02 2024/11/19-15:16:58.980601 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00063/att-02/compact.done) = No such file or directory: 0.037 sec on http probe failed: 5, giveup 2024/11/19-15:16:58.980614 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00063/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.338 (submit 296.134 ms wait 0.037 s), input 525 B, send shutdown 2024/11/19-15:16:58.983529 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00064/att-02 2024/11/19-15:16:58.985651 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00062/att-02 2024/11/19-15:16:58.988439 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00063/att-02 2024/11/19-15:16:59.033734 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4528: 32057 keys, 33524626 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.033789 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619033767, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4528, "file_size": 33524626, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32602688, "index_size": 921088, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602668, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000618, "oldest_key_time": 0, "file_creation_time": 1732000618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4528, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.036072 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-00 2024/11/19-15:16:59.036191 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00061/att-00/compact.done) = No such file or directory: 1.573 sec on http probe failed: 1, retry 2024/11/19-15:16:59.059336 139851137550080 [dcompact_etcd.cc:1773] job-00064/att-02: CleanFiles 0.076 sec : 0.000 2024/11/19-15:16:59.059413 139851148035840 [dcompact_etcd.cc:1773] job-00062/att-02: CleanFiles 0.074 sec : 0.000 2024/11/19-15:16:59.059470 139851158521600 [dcompact_etcd.cc:1773] job-00063/att-02: CleanFiles 0.071 sec : 0.000 2024/11/19-15:16:59.061181 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00064) = success 2024/11/19-15:16:59.061189 139851137550080 [dcompact_etcd.cc:1773] job-00064/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:16:59.061209 139851137550080 (Original Log Time 2024/11/19-15:16:57.937521) EVENT_LOG_v1 {"time_micros": 1732000617937515, "job": 64, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4082], "files_L3": [3802, 3953], "score": 0.314151, "input_data_size": 72185, "oldest_snapshot_seqno": 113934572} 2024/11/19-15:16:59.061217 139851137550080 [compaction_job.cc:2437] [default] [JOB 64] Compacting 1@2 + 2@3 files to L3, score 0.31, subcompactions 1 : 1 2024/11/19-15:16:59.061225 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 62 Base level 2, inputs: [4082(30KB)], [3802(17KB) 3953(22KB)] 2024/11/19-15:16:59.061468 139851137550080 [compaction_job.cc:703] job-00064: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:16:59.061756 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:16:59.061865 139851137550080 (Original Log Time 2024/11/19-15:16:59.061259) EVENT_LOG_v1 {"time_micros": 1732000619061238, "job": 64, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4082], "files_L3": [3802, 3953], "score": 0.314151, "input_data_size": 72185, "oldest_snapshot_seqno": 113934572} 2024/11/19-15:16:59.061871 139851137550080 (Original Log Time 2024/11/19-15:16:59.061543) [compaction_job.cc:2022] [default] [JOB 64] Compacted 1@2 + 2@3 files to L3 => 0 bytes 2024/11/19-15:16:59.061874 139851137550080 (Original Log Time 2024/11/19-15:16:59.061731) [compaction_job.cc:1224] [default] compacted to: files[5 25 15 44 0 0 0] max score 0.31 (8 files need compaction), MB/sec: 335.7 rd, 0.0 wr, level 3, files in(1, 2) out(0 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(2.3) write-amplify(0.0) OK, records in: 362, records dropped: 362 output_compression: Snappy 2024/11/19-15:16:59.061881 139851137550080 (Original Log Time 2024/11/19-15:16:59.061750) EVENT_LOG_v1 {"time_micros": 1732000619061740, "cf": "default", "job": 64, "event": "compaction_finished", "compaction_time_micros": 215, "compaction_time_cpu_micros": 199, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 362, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 25, 15, 44, 0, 0, 0]} 2024/11/19-15:16:59.061982 139851137550080 [compaction_job.cc:2437] [default] [JOB 65] Compacting 1@2 files to L3, score 0.31, subcompactions 1 : 1 2024/11/19-15:16:59.061991 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 63 Base level 2, inputs: [4457(63MB)] 2024/11/19-15:16:59.062635 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00062) = success 2024/11/19-15:16:59.062648 139851148035840 [dcompact_etcd.cc:1773] job-00062/att-03: CleanFiles 0.003 sec : 0.000 2024/11/19-15:16:59.062667 139851148035840 (Original Log Time 2024/11/19-15:16:57.911256) EVENT_LOG_v1 {"time_micros": 1732000617911247, "job": 62, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [4445], "score": 0.314151, "input_data_size": 3396, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:59.062673 139851148035840 [compaction_job.cc:2437] [default] [JOB 62] Compacting 1@1 files to L2, score 0.31, subcompactions 1 : 1 2024/11/19-15:16:59.062679 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 62 Base level 1, inputs: [4445(3396B)] 2024/11/19-15:16:59.062870 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004530.sst) /storage/mytopling/..//tmp/Topling-82038-Na7use 2024/11/19-15:16:59.063007 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004530.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:16:59.063212 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004530.sst): first pass time = 0.00's, 0.711'MB/sec 2024/11/19-15:16:59.063258 139834056247040 [fixed_len_key_index.cc:363] FixedLenKeyIndex build_cache Finish: num_keys = 4, fixlen = 27, pref_len = 4, cplen = 0, time = 0.000 sec 2024/11/19-15:16:59.063322 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004530.sst): index pass time = 0.00's, 0.364'MB/sec index type = FixedLenKeyIndex usrkeys = 4 min-keylen = 27 max-keylen = 27 prefix = 4 subcplen = 0 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 27.00 avg-zkey =2256.00 2024/11/19-15:16:59.063376 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004530.sst): old prealloc_size = 3735, real_size = 9664 2024/11/19-15:16:59.063395 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004530.sst): index type = FixedLenKeyIndex , store type = MixedLenBlobStore usrkeys = 4 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 27.00 avg-zkey =2256.00 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 20.00 avg-zval = 108.00 2024/11/19-15:16:59.063542 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004530.sst): second pass time = 0.00's,1111.111'MB/sec, value only(33.9% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 67.202'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 581.931'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.085'MB/sec write SST data time = 0.00's, 80.704'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 1111.111'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 4 avg-key = 39.00 avg-zkey = 2256.00 avg-val = 20.00 avg-zval = 108.00 usrkeys = 4 avg-key = 31.00 avg-zkey = 2256.00 avg-val = 20.00 avg-zval = 108.00 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 3 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.0137 value = 0.1852 dict = 0.00 all = 0.0224 } Zip/UnZip{ index = 72.7742 value = 5.4000 dict = 0.00 all = 44.7161 } ---------------------------- total value len = 2.455844 GB avg = 0.430 KB (by entry num) total key len = 0.141617 GB avg = 0.025 KB total ukey len = 0.095883 GB avg = 0.017 KB total ukey num = 0.005716668 Billion total entry num = 0.005716668 Billion write speed all = 20.891174215 MB/sec (with seq num) write speed all = 20.523344448 MB/sec (without seq num) 2024/11/19-15:16:59.063847 139851148035840 [compaction_job.cc:1948] [default] [JOB 62] Generated table #4530: 4 keys, 10553 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.063883 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000619063860, "cf_name": "default", "job": 62, "event": "table_file_creation", "file_number": 4530, "file_size": 10553, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 432, "index_size": 9024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 156, "raw_average_key_size": 39, "raw_value_size": 80, "raw_average_value_size": 20, "num_data_blocks": 1, "num_entries": 4, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 39, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-FixedLenKeyIndex-MixedLenBlobStore", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4530, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:59.063991 139851148035840 [compaction_job.cc:703] job-00062: subcompact[0], size: 0.000011 G, files: /storage/mytopling/.rocksdb [004530] 2024/11/19-15:16:59.064094 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00063) = success 2024/11/19-15:16:59.064106 139851158521600 [dcompact_etcd.cc:1773] job-00063/att-03: CleanFiles 0.005 sec : 0.000 2024/11/19-15:16:59.064124 139851158521600 (Original Log Time 2024/11/19-15:16:57.928858) EVENT_LOG_v1 {"time_micros": 1732000617928851, "job": 63, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [4448], "score": 0.314151, "input_data_size": 3588, "oldest_snapshot_seqno": 113934572} 2024/11/19-15:16:59.064131 139851158521600 [compaction_job.cc:2437] [default] [JOB 63] Compacting 1@1 files to L2, score 0.31, subcompactions 1 : 1 2024/11/19-15:16:59.064137 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 62 Base level 1, inputs: [4448(3588B)] 2024/11/19-15:16:59.064286 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004531.sst) /storage/mytopling/..//tmp/Topling-82038-Kxpv0o 2024/11/19-15:16:59.064350 139851148035840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16273899 2024/11/19-15:16:59.064390 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004531.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:16:59.064414 139851148035840 (Original Log Time 2024/11/19-15:16:59.062705) EVENT_LOG_v1 {"time_micros": 1732000619062687, "job": 62, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [4445], "score": 0.314151, "input_data_size": 3396, "oldest_snapshot_seqno": -1} 2024/11/19-15:16:59.064420 139851148035840 (Original Log Time 2024/11/19-15:16:59.064182) [compaction_job.cc:2022] [default] [JOB 62] Compacted 1@1 files to L2 => 10553 bytes 2024/11/19-15:16:59.064423 139851148035840 (Original Log Time 2024/11/19-15:16:59.064330) [compaction_job.cc:1224] [default] compacted to: files[5 24 16 44 0 0 0] max score 0.31 (8 files need compaction), MB/sec: 2.6 rd, 8.2 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(4.1) write-amplify(3.1) OK, records in: 4, records dropped: 0 output_compression: Snappy 2024/11/19-15:16:59.064430 139851148035840 (Original Log Time 2024/11/19-15:16:59.064346) EVENT_LOG_v1 {"time_micros": 1732000619064338, "cf": "default", "job": 62, "event": "compaction_finished", "compaction_time_micros": 1293, "compaction_time_cpu_micros": 934, "output_level": 2, "num_output_files": 1, "total_output_size": 10553, "num_input_records": 4, "num_output_records": 4, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 24, 16, 44, 0, 0, 0]} 2024/11/19-15:16:59.064486 139851148035840 [compaction_job.cc:2437] [default] [JOB 66] Compacting 1@2 files to L3, score 0.31, subcompactions 1 : 1 2024/11/19-15:16:59.064493 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 64 Base level 2, inputs: [4498(62MB)] 2024/11/19-15:16:59.064502 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004531.sst): index pass time = 0.00's, 0.183'MB/sec index type = UintIndex_AllOne usrkeys = 4 min-keylen = 4 max-keylen = 4 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 40.00 2024/11/19-15:16:59.064564 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004531.sst): first pass time = 0.00's, 1.948'MB/sec 2024/11/19-15:16:59.064592 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004531.sst): old prealloc_size = 3946, real_size = 1184 2024/11/19-15:16:59.064605 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004531.sst): index type = UintIndex_AllOne , store type = MixedLenBlobStore usrkeys = 4 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 40.00 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 115.25 avg-zval = 204.00 2024/11/19-15:16:59.064739 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004531.sst): second pass time = 0.00's,6229.730'MB/sec, value only(87.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 15.558'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 12.184'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.169'MB/sec write SST data time = 0.00's, 18.367'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 6229.730'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 4 avg-key = 16.00 avg-zkey = 48.00 avg-val = 115.25 avg-zval = 204.00 usrkeys = 4 avg-key = 8.00 avg-zkey = 48.00 avg-val = 115.25 avg-zval = 204.00 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 4 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.1667 value = 0.5650 dict = 0.00 all = 0.2494 } Zip/UnZip{ index = 6.0000 value = 1.7701 dict = 0.00 all = 4.0095 } ---------------------------- total value len = 2.455844 GB avg = 0.430 KB (by entry num) total key len = 0.141617 GB avg = 0.025 KB total ukey len = 0.095883 GB avg = 0.017 KB total ukey num = 0.005716672 Billion total entry num = 0.005716672 Billion write speed all = 20.890977115 MB/sec (with seq num) write speed all = 20.523150635 MB/sec (without seq num) 2024/11/19-15:16:59.065103 139851158521600 [compaction_job.cc:1948] [default] [JOB 63] Generated table #4531: 4 keys, 2105 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.065136 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000619065117, "cf_name": "default", "job": 63, "event": "table_file_creation", "file_number": 4531, "file_size": 2105, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 816, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 64, "raw_average_key_size": 16, "raw_value_size": 461, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 4, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-MixedLenBlobStore", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4531, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:16:59.065219 139851158521600 [compaction_job.cc:703] job-00063: subcompact[0], size: 0.000002 G, files: /storage/mytopling/.rocksdb [004531] 2024/11/19-15:16:59.065547 139851158521600 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777215 2024/11/19-15:16:59.065620 139851158521600 (Original Log Time 2024/11/19-15:16:59.064161) EVENT_LOG_v1 {"time_micros": 1732000619064144, "job": 63, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [4448], "score": 0.314151, "input_data_size": 3588, "oldest_snapshot_seqno": 113934572} 2024/11/19-15:16:59.065626 139851158521600 (Original Log Time 2024/11/19-15:16:59.065388) [compaction_job.cc:2022] [default] [JOB 63] Compacted 1@1 files to L2 => 2105 bytes 2024/11/19-15:16:59.065629 139851158521600 (Original Log Time 2024/11/19-15:16:59.065528) [compaction_job.cc:1224] [default] compacted to: files[5 23 17 44 0 0 0] max score 0.31 (8 files need compaction), MB/sec: 3.4 rd, 2.0 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.6) write-amplify(0.6) OK, records in: 4, records dropped: 0 output_compression: Snappy 2024/11/19-15:16:59.065636 139851158521600 (Original Log Time 2024/11/19-15:16:59.065543) EVENT_LOG_v1 {"time_micros": 1732000619065535, "cf": "default", "job": 63, "event": "compaction_finished", "compaction_time_micros": 1064, "compaction_time_cpu_micros": 773, "output_level": 2, "num_output_files": 1, "total_output_size": 2105, "num_input_records": 4, "num_output_records": 4, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 23, 17, 44, 0, 0, 0]} 2024/11/19-15:16:59.065692 139851158521600 [compaction_job.cc:2437] [default] [JOB 67] Compacting 1@2 files to L3, score 0.31, subcompactions 1 : 1 2024/11/19-15:16:59.065700 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 65 Base level 2, inputs: [4504(61MB)] 2024/11/19-15:16:59.066047 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4529: 32053 keys, 33525010 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.066083 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619066065, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4529, "file_size": 33525010, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32599744, "index_size": 924416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32599691, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4529, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.097711 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4532: 32152 keys, 33525202 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.097757 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619097737, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4532, "file_size": 33525202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32697728, "index_size": 826624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643040, "raw_average_key_size": 20, "raw_value_size": 32697694, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32152, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4532, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.129502 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4533: 32055 keys, 33525138 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.129551 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619129530, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4533, "file_size": 33525138, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32598848, "index_size": 925440, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32598840, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4533, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.171886 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4534: 32057 keys, 33525266 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.171945 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619171916, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4534, "file_size": 33525266, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32600064, "index_size": 924352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32600031, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4534, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.214501 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4535: 32060 keys, 33526098 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.214549 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619214529, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4535, "file_size": 33526098, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32606208, "index_size": 919040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32606206, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4535, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.290412 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4536: 32053 keys, 33524754 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.290462 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619290442, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4536, "file_size": 33524754, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32598080, "index_size": 925824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32598040, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4536, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.322268 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4537: 32056 keys, 33525138 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.322316 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619322295, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4537, "file_size": 33525138, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32600256, "index_size": 924032, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32600205, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4537, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.358135 139851137550080 [dcompact_etcd.cc:1163] job-00065/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.296 sec, input = 64.000 MiB 2024/11/19-15:16:59.373522 139851148035840 [dcompact_etcd.cc:1163] job-00066/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.309 sec, input = 64.001 MiB 2024/11/19-15:16:59.373569 139851158521600 [dcompact_etcd.cc:1163] job-00067/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.307 sec, input = 62.824 MiB 2024/11/19-15:16:59.437873 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4538: 32066 keys, 33524434 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.437935 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619437902, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4538, "file_size": 33524434, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32610688, "index_size": 912896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641320, "raw_average_key_size": 20, "raw_value_size": 32610682, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32066, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4538, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.480627 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4539: 32055 keys, 33524690 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.480677 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619480656, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4539, "file_size": 33524690, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32598464, "index_size": 925376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32598434, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4539, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.512437 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4540: 32056 keys, 33524946 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.512486 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619512465, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4540, "file_size": 33524946, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32600640, "index_size": 923456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32600615, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4540, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.544624 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4541: 32083 keys, 33525202 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.544688 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619544659, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4541, "file_size": 33525202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32627904, "index_size": 896448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641660, "raw_average_key_size": 20, "raw_value_size": 32627885, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32083, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4541, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.592507 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4542: 32057 keys, 33524882 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.592562 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619592541, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4542, "file_size": 33524882, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32599936, "index_size": 924096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32599882, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4542, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.669972 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4543: 32057 keys, 33525330 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.670032 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619670010, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4543, "file_size": 33525330, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32601152, "index_size": 923328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601107, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4543, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.702731 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4544: 32112 keys, 33524882 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.702787 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619702766, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4544, "file_size": 33524882, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32659584, "index_size": 864448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642240, "raw_average_key_size": 20, "raw_value_size": 32659545, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32112, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4544, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.812142 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4545: 32053 keys, 33525138 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.812200 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619812177, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4545, "file_size": 33525138, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32599296, "index_size": 924992, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32599240, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4545, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.844594 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4546: 32152 keys, 33525202 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.844643 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619844623, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4546, "file_size": 33525202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32697792, "index_size": 826560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643040, "raw_average_key_size": 20, "raw_value_size": 32697772, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32152, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4546, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.878558 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4547: 32059 keys, 33525330 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.878606 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619878586, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4547, "file_size": 33525330, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32604480, "index_size": 920000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32604448, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4547, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.910396 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4548: 32078 keys, 33524370 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.910444 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619910424, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4548, "file_size": 33524370, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32624256, "index_size": 899264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641560, "raw_average_key_size": 20, "raw_value_size": 32624241, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32078, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4548, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.948772 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4549: 32057 keys, 33525202 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.948822 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619948803, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4549, "file_size": 33525202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32603008, "index_size": 921344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602951, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4549, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:16:59.986543 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4550: 32060 keys, 33524754 bytes (need compaction), temperature: kUnknown 2024/11/19-15:16:59.986595 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000619986574, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4550, "file_size": 33524754, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32603904, "index_size": 920000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32603898, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4550, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.018954 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4551: 32111 keys, 33524626 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.019003 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620018983, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4551, "file_size": 33524626, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32655232, "index_size": 868544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642220, "raw_average_key_size": 20, "raw_value_size": 32655214, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32111, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000619, "oldest_key_time": 0, "file_creation_time": 1732000619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4551, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.084907 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4552: 32058 keys, 33525330 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.085020 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620084978, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4552, "file_size": 33525330, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32603328, "index_size": 921152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603321, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4552, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.119228 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-02 2024/11/19-15:17:00.119359 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00051/att-02/compact.done) = No such file or directory: 8.890 sec on http probe failed: 4, retry 2024/11/19-15:17:00.160537 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4553: 32061 keys, 33525202 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.160588 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620160566, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4553, "file_size": 33525202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32603520, "index_size": 920832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641220, "raw_average_key_size": 20, "raw_value_size": 32603491, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32061, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4553, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.192229 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4554: 32152 keys, 33524882 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.192280 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620192258, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4554, "file_size": 33524882, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32696768, "index_size": 827264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643040, "raw_average_key_size": 20, "raw_value_size": 32696720, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32152, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4554, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.223964 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4555: 32057 keys, 33524434 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.224016 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620223995, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4555, "file_size": 33524434, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32601792, "index_size": 921792, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601780, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4555, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.256869 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4556: 32058 keys, 33525074 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.256933 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620256901, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4556, "file_size": 33525074, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32603840, "index_size": 920384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603779, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4556, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.299504 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-00 2024/11/19-15:17:00.299574 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00061/att-00/compact.done) = No such file or directory: 2.836 sec on http probe failed: 2, retry 2024/11/19-15:17:00.303104 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4557: 32150 keys, 33524242 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.303146 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620303125, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4557, "file_size": 33524242, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32696960, "index_size": 826432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32696925, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4557, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.353496 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4558: 32055 keys, 33524562 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.353552 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620353529, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4558, "file_size": 33524562, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32601280, "index_size": 922432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32601265, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4558, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.387494 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4559: 32057 keys, 33524882 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.387558 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620387528, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4559, "file_size": 33524882, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32601152, "index_size": 922880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601105, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4559, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.419320 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4560: 32126 keys, 33525202 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.419370 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620419350, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4560, "file_size": 33525202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32673472, "index_size": 850880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642520, "raw_average_key_size": 20, "raw_value_size": 32673435, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32126, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4560, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.455859 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4561: 32057 keys, 33525138 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.455911 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620455889, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4561, "file_size": 33525138, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32602368, "index_size": 921920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602308, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4561, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.533752 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4562: 32050 keys, 33525330 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.533812 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620533786, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4562, "file_size": 33525330, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32597248, "index_size": 927232, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641000, "raw_average_key_size": 20, "raw_value_size": 32597197, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32050, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4562, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.637071 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4563: 32150 keys, 33524882 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.637130 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620637108, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4563, "file_size": 33524882, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32697536, "index_size": 826496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32697508, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4563, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.668328 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4564: 32054 keys, 33524754 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.668379 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620668358, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4564, "file_size": 33524754, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32597696, "index_size": 926208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641080, "raw_average_key_size": 20, "raw_value_size": 32597684, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4564, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.686388 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-01 2024/11/19-15:17:00.686486 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00054/att-01/compact.done) = No such file or directory: 13.024 sec on http probe failed: 5, giveup 2024/11/19-15:17:00.686512 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00054/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 244.368 sec, real 13.329 (submit 296.315 ms wait 13.024 s), input 244.368 MiB, send shutdown 2024/11/19-15:17:00.691661 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-01 2024/11/19-15:17:00.691908 139851169007360 [dcompact_etcd.cc:1773] job-00054/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:00.701667 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4565: 32055 keys, 33525074 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.701711 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620701691, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4565, "file_size": 33525074, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32599872, "index_size": 924352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32599861, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4565, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.733691 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4566: 32059 keys, 33524434 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.733749 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620733726, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4566, "file_size": 33524434, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32604544, "index_size": 919040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32604512, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4566, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.854376 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4567: 32051 keys, 33524498 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.854431 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620854409, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4567, "file_size": 33524498, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32597760, "index_size": 925888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641020, "raw_average_key_size": 20, "raw_value_size": 32597750, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32051, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4567, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.892926 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4568: 32055 keys, 33525138 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.892996 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620892966, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4568, "file_size": 33525138, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32600320, "index_size": 923968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32600271, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4568, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.924564 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4569: 32061 keys, 33525202 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.924616 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620924595, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4569, "file_size": 33525202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32605312, "index_size": 919040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641220, "raw_average_key_size": 20, "raw_value_size": 32605260, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32061, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4569, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.952621 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-00 2024/11/19-15:17:00.952746 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00067/att-00/compact.done) = No such file or directory: 1.571 sec on http probe failed: 1, retry 2024/11/19-15:17:00.967431 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-00 2024/11/19-15:17:00.967533 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00065/att-00/compact.done) = No such file or directory: 1.600 sec on http probe failed: 1, retry 2024/11/19-15:17:00.971943 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4570: 32052 keys, 33524946 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:00.971993 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000620971971, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4570, "file_size": 33524946, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32598720, "index_size": 925376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641040, "raw_average_key_size": 20, "raw_value_size": 32598694, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32052, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4570, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:00.982651 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-00 2024/11/19-15:17:00.982722 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00066/att-00/compact.done) = No such file or directory: 1.600 sec on http probe failed: 1, retry 2024/11/19-15:17:01.006558 139851169007360 [dcompact_etcd.cc:1163] job-00054/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.315 sec, input = 244.368 MiB 2024/11/19-15:17:01.013322 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4571: 32148 keys, 33525010 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:01.013383 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000621013354, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4571, "file_size": 33525010, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32695424, "index_size": 828736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642960, "raw_average_key_size": 20, "raw_value_size": 32695396, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32148, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000620, "oldest_key_time": 0, "file_creation_time": 1732000620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4571, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:01.071143 139849937979136 [compaction_job.cc:1948] [default] [JOB 60] Generated table #4572: 9893 keys, 10358289 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:01.071194 139849937979136 EVENT_LOG_v1 {"time_micros": 1732000621071171, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 4572, "file_size": 10358289, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 10061440, "index_size": 296000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 197860, "raw_average_key_size": 20, "raw_value_size": 10061403, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 9893, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000621, "oldest_key_time": 0, "file_creation_time": 1732000621, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4572, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:17:01.071289 139851127064320 [compaction_job.cc:703] job-00060: subcompact[0], size: 0.327534 G, files: /storage/mytopling/.rocksdb [004473,004475,004476,004477,004478,004480,004482,004484,004486,004487,004488,004490,004491] 2024/11/19-15:17:01.071313 139851127064320 [compaction_job.cc:703] job-00060: subcompact[1], size: 0.243498 G, files: /storage/mytopling/.rocksdb [004472,004481,004483,004492,004494,004496,004499,004503,004506] 2024/11/19-15:17:01.071349 139851127064320 [compaction_job.cc:703] job-00060: subcompact[2], size: 2.302117 G, files: /storage/mytopling/.rocksdb [004474,004479,004485,004489,004493,004495,004497,004500,004505,004509,004510,004511,004512,004513,004514,004515,004516,004517,004518,004519,004520,004521,004522,004523,004524,004525,004526,004527,004528,004529,004532,004533,004534,004535,004536,004537,004538,004539,004540,004541,004542,004543,004544,004545,004546,004547,004548,004549,004550,004551,004552,004553,004554,004555,004556,004557,004558,004559,004560,004561,004562,004563,004564,004565,004566,004567,004568,004569,004570,004571,004572] 2024/11/19-15:17:01.169640 139851127064320 (Original Log Time 2024/11/19-15:16:56.081514) EVENT_LOG_v1 {"time_micros": 1732000616081504, "job": 60, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [4444, 4436, 4432, 4428, 4340], "files_L1": [4449, 4450, 4447, 4452, 4453, 4454, 4455, 4446, 4451, 4456, 4458, 4459, 4460, 4461, 4463, 4464, 4465, 4466, 4467, 4468, 4469, 4470, 4471], "score": 3.04095, "input_data_size": 3185753944, "oldest_snapshot_seqno": 113894662} 2024/11/19-15:17:01.169655 139851127064320 (Original Log Time 2024/11/19-15:17:01.168931) [compaction_job.cc:2022] [default] [JOB 60] Compacted 5@0 + 23@1 files to L1 => 2873149513 bytes 2024/11/19-15:17:01.169659 139851127064320 (Original Log Time 2024/11/19-15:17:01.169377) [compaction_job.cc:1224] [default] compacted to: files[0 93 17 44 0 0 0] max score 5.36 (100 files need compaction), MB/sec: 638.4 rd, 575.8 wr, level 1, files in(5, 23) out(93 +0 blob) MB in(2571.5, 466.7 +0.0 blob) out(2740.0 +0.0 blob), read-write-amplify(2.2) write-amplify(1.1) OK, records in: 8192524, records dropped: 0 output_compression: Snappy 2024/11/19-15:17:01.169674 139851127064320 (Original Log Time 2024/11/19-15:17:01.169431) EVENT_LOG_v1 {"time_micros": 1732000621169391, "cf": "default", "job": 60, "event": "compaction_finished", "compaction_time_micros": 4989840, "compaction_time_cpu_micros": 4922788, "output_level": 1, "num_output_files": 93, "total_output_size": 2873149513, "num_input_records": 8192524, "num_output_records": 8192524, "num_subcompactions": 3, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 93, 17, 44, 0, 0, 0]} 2024/11/19-15:17:01.169759 139851127064320 [compaction_job.cc:2437] [default] [JOB 68] Compacting 4@1 files to L2, score 5.36, subcompactions 1 : 1 2024/11/19-15:17:01.169768 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 66 Base level 1, inputs: [4473(5573B) 4475(27KB) 4476(31MB) 4477(31MB)] 2024/11/19-15:17:01.449371 139851127064320 [dcompact_etcd.cc:1163] job-00068/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.279 sec, input = 63.642 MiB 2024/11/19-15:17:01.567013 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-00 2024/11/19-15:17:01.567117 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00061/att-00/compact.done) = No such file or directory: 4.100 sec on http probe failed: 3, retry 2024/11/19-15:17:02.129117 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-02 2024/11/19-15:17:02.129212 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00051/att-02/compact.done) = No such file or directory: 10.899 sec on http probe failed: 5, giveup 2024/11/19-15:17:02.129231 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00051/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 114.819 sec, real 11.177 (submit 267.863 ms wait 10.899 s), input 114.819 MiB, send shutdown 2024/11/19-15:17:02.137012 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00051/att-02 2024/11/19-15:17:02.218177 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-00 2024/11/19-15:17:02.218249 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00067/att-00/compact.done) = No such file or directory: 2.836 sec on http probe failed: 2, retry 2024/11/19-15:17:02.242588 139851179493120 [dcompact_etcd.cc:1773] job-00051/att-02: CleanFiles 0.106 sec : 0.000 2024/11/19-15:17:02.242993 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00051) = success 2024/11/19-15:17:02.243002 139851179493120 [dcompact_etcd.cc:1773] job-00051/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:02.243024 139851179493120 (Original Log Time 2024/11/19-15:16:28.521859) EVENT_LOG_v1 {"time_micros": 1732000588521854, "job": 51, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4433, 4434], "score": 2.42299, "input_data_size": 116971419, "oldest_snapshot_seqno": -1} 2024/11/19-15:17:02.243032 139851179493120 [compaction_job.cc:2437] [default] [JOB 51] Compacting 2@2 files to L3, score 2.42, subcompactions 1 : 1 2024/11/19-15:17:02.243039 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 49 Base level 2, inputs: [4433(62MB) 4434(49MB)] 2024/11/19-15:17:02.252352 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-00 2024/11/19-15:17:02.252409 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00065/att-00/compact.done) = No such file or directory: 2.889 sec on http probe failed: 2, retry 2024/11/19-15:17:02.261357 139851179493120 [compaction_job.cc:703] job-00051: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:17:02.270498 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-00 2024/11/19-15:17:02.270550 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00066/att-00/compact.done) = No such file or directory: 2.889 sec on http probe failed: 2, retry 2024/11/19-15:17:02.273628 139851179493120 (Original Log Time 2024/11/19-15:17:02.243072) EVENT_LOG_v1 {"time_micros": 1732000622243052, "job": 51, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4433, 4434], "score": 2.42299, "input_data_size": 116971419, "oldest_snapshot_seqno": -1} 2024/11/19-15:17:02.273637 139851179493120 (Original Log Time 2024/11/19-15:17:02.273266) [compaction_job.cc:2022] [default] [JOB 51] Compacted 2@2 files to L3 => 0 bytes 2024/11/19-15:17:02.273640 139851179493120 (Original Log Time 2024/11/19-15:17:02.273520) [compaction_job.cc:1224] [default] compacted to: files[0 93 15 44 0 0 0] max score 5.23 (96 files need compaction), MB/sec: 6392.9 rd, 0.0 wr, level 3, files in(2, 0) out(0 +0 blob) MB in(111.6, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 180000, records dropped: 180000 output_compression: Snappy 2024/11/19-15:17:02.273658 139851179493120 (Original Log Time 2024/11/19-15:17:02.273542) EVENT_LOG_v1 {"time_micros": 1732000622273531, "cf": "default", "job": 51, "event": "compaction_finished", "compaction_time_micros": 18297, "compaction_time_cpu_micros": 18194, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 180000, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 93, 15, 44, 0, 0, 0]} 2024/11/19-15:17:02.314203 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004394.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:02.314238 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000622314234, "job": 51, "event": "table_file_deletion", "file_number": 4394} 2024/11/19-15:17:02.322483 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004396.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:02.322513 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000622322510, "job": 51, "event": "table_file_deletion", "file_number": 4396} 2024/11/19-15:17:02.340336 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004398.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:02.340368 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000622340364, "job": 51, "event": "table_file_deletion", "file_number": 4398} 2024/11/19-15:17:02.363236 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004400.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:02.363288 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000622363282, "job": 51, "event": "table_file_deletion", "file_number": 4400} 2024/11/19-15:17:02.370370 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004426.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:02.370404 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000622370400, "job": 51, "event": "table_file_deletion", "file_number": 4426} 2024/11/19-15:17:02.474776 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004430.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:02.474828 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000622474822, "job": 51, "event": "table_file_deletion", "file_number": 4430} 2024/11/19-15:17:02.475067 139851179493120 [compaction_job.cc:2437] [default] [JOB 69] Compacting 4@1 files to L2, score 5.23, subcompactions 1 : 1 2024/11/19-15:17:02.475082 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 67 Base level 1, inputs: [4478(31MB) 4480(31MB) 4482(31MB) 4484(31MB)] 2024/11/19-15:17:02.759101 139851179493120 [dcompact_etcd.cc:1163] job-00069/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.283 sec, input = 127.237 MiB 2024/11/19-15:17:02.832873 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-00 2024/11/19-15:17:02.832961 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00061/att-00/compact.done) = No such file or directory: 5.367 sec on http probe failed: 4, retry 2024/11/19-15:17:03.048573 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-00 2024/11/19-15:17:03.048677 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00068/att-00/compact.done) = No such file or directory: 1.591 sec on http probe failed: 1, retry 2024/11/19-15:17:03.481540 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-00 2024/11/19-15:17:03.481648 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00067/att-00/compact.done) = No such file or directory: 4.101 sec on http probe failed: 3, retry 2024/11/19-15:17:03.541083 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-00 2024/11/19-15:17:03.541171 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00065/att-00/compact.done) = No such file or directory: 4.174 sec on http probe failed: 3, retry 2024/11/19-15:17:03.558822 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-00 2024/11/19-15:17:03.558902 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00066/att-00/compact.done) = No such file or directory: 4.177 sec on http probe failed: 3, retry 2024/11/19-15:17:04.100216 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-00 2024/11/19-15:17:04.100301 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00061/att-00/compact.done) = No such file or directory: 6.633 sec on http probe failed: 5, giveup 2024/11/19-15:17:04.100321 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00061/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 62.907 sec, real 6.922 (submit 279.616 ms wait 6.633 s), input 62.907 MiB, send shutdown 2024/11/19-15:17:04.106278 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-00 2024/11/19-15:17:04.149926 139850978166528 [dcompact_etcd.cc:1773] job-00061/att-00: CleanFiles 0.044 sec : 0.000 2024/11/19-15:17:04.329485 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-00 2024/11/19-15:17:04.329571 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00068/att-00/compact.done) = No such file or directory: 2.872 sec on http probe failed: 2, retry 2024/11/19-15:17:04.431404 139850978166528 [dcompact_etcd.cc:1163] job-00061/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 62.907 MiB 2024/11/19-15:17:04.752576 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-00 2024/11/19-15:17:04.752647 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00067/att-00/compact.done) = No such file or directory: 5.365 sec on http probe failed: 4, retry 2024/11/19-15:17:04.827237 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-00 2024/11/19-15:17:04.827301 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00065/att-00/compact.done) = No such file or directory: 5.463 sec on http probe failed: 4, retry 2024/11/19-15:17:04.847648 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-00 2024/11/19-15:17:04.847704 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00066/att-00/compact.done) = No such file or directory: 5.465 sec on http probe failed: 4, retry 2024/11/19-15:17:05.608521 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-00 2024/11/19-15:17:05.608614 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00068/att-00/compact.done) = No such file or directory: 4.153 sec on http probe failed: 3, retry 2024/11/19-15:17:05.948181 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-00 2024/11/19-15:17:05.948275 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00069/att-00/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:17:06.013180 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-01 2024/11/19-15:17:06.013276 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00061/att-01/compact.done) = No such file or directory: 1.573 sec on http probe failed: 1, retry 2024/11/19-15:17:06.014466 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-00 2024/11/19-15:17:06.014531 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00067/att-00/compact.done) = No such file or directory: 6.636 sec on http probe failed: 5, giveup 2024/11/19-15:17:06.014547 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00067/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 62.824 sec, real 6.948 (submit 307.432 ms wait 6.636 s), input 62.824 MiB, send shutdown 2024/11/19-15:17:06.015331 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-02 2024/11/19-15:17:06.015390 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00054/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:17:06.022494 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-00 2024/11/19-15:17:06.022809 139851158521600 [dcompact_etcd.cc:1773] job-00067/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:06.112487 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-00 2024/11/19-15:17:06.112569 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00065/att-00/compact.done) = No such file or directory: 6.749 sec on http probe failed: 5, giveup 2024/11/19-15:17:06.112586 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00065/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 64.000 sec, real 7.050 (submit 295.698 ms wait 6.749 s), input 64.000 MiB, send shutdown 2024/11/19-15:17:06.121062 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-00 2024/11/19-15:17:06.121392 139851137550080 [dcompact_etcd.cc:1773] job-00065/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:06.136646 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-00 2024/11/19-15:17:06.136725 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00066/att-00/compact.done) = No such file or directory: 6.754 sec on http probe failed: 5, giveup 2024/11/19-15:17:06.136749 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00066/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 64.001 sec, real 7.072 (submit 308.640 ms wait 6.754 s), input 64.001 MiB, send shutdown 2024/11/19-15:17:06.145997 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-00 2024/11/19-15:17:06.146252 139851148035840 [dcompact_etcd.cc:1773] job-00066/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:06.323249 139851158521600 [dcompact_etcd.cc:1163] job-00067/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.300 sec, input = 62.824 MiB 2024/11/19-15:17:06.427581 139851137550080 [dcompact_etcd.cc:1163] job-00065/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.306 sec, input = 64.000 MiB 2024/11/19-15:17:06.449497 139851148035840 [dcompact_etcd.cc:1163] job-00066/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.303 sec, input = 64.001 MiB 2024/11/19-15:17:06.887134 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-00 2024/11/19-15:17:06.887212 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00068/att-00/compact.done) = No such file or directory: 5.432 sec on http probe failed: 4, retry 2024/11/19-15:17:07.281742 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-01 2024/11/19-15:17:07.281821 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00061/att-01/compact.done) = No such file or directory: 2.840 sec on http probe failed: 2, retry 2024/11/19-15:17:07.899328 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-01 2024/11/19-15:17:07.899445 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00067/att-01/compact.done) = No such file or directory: 1.571 sec on http probe failed: 1, retry 2024/11/19-15:17:07.960537 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-00 2024/11/19-15:17:07.960621 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00069/att-00/compact.done) = No such file or directory: 5.189 sec on http probe failed: 2, retry 2024/11/19-15:17:08.020604 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-02 2024/11/19-15:17:08.020687 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00054/att-02/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:17:08.036224 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-01 2024/11/19-15:17:08.036287 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00065/att-01/compact.done) = No such file or directory: 1.600 sec on http probe failed: 1, retry 2024/11/19-15:17:08.054718 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-01 2024/11/19-15:17:08.054779 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00066/att-01/compact.done) = No such file or directory: 1.600 sec on http probe failed: 1, retry 2024/11/19-15:17:08.166352 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-00 2024/11/19-15:17:08.166431 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00068/att-00/compact.done) = No such file or directory: 6.711 sec on http probe failed: 5, giveup 2024/11/19-15:17:08.166448 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00068/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 63.642 sec, real 6.996 (submit 279.014 ms wait 6.711 s), input 63.642 MiB, send shutdown 2024/11/19-15:17:08.174495 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-00 2024/11/19-15:17:08.239264 139851127064320 [dcompact_etcd.cc:1773] job-00068/att-00: CleanFiles 0.065 sec : 0.000 2024/11/19-15:17:08.510270 139851127064320 [dcompact_etcd.cc:1163] job-00068/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.271 sec, input = 63.642 MiB 2024/11/19-15:17:08.545700 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-01 2024/11/19-15:17:08.545775 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00061/att-01/compact.done) = No such file or directory: 4.109 sec on http probe failed: 3, retry 2024/11/19-15:17:09.161895 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-01 2024/11/19-15:17:09.161984 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00067/att-01/compact.done) = No such file or directory: 2.833 sec on http probe failed: 2, retry 2024/11/19-15:17:09.325681 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-01 2024/11/19-15:17:09.325757 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00065/att-01/compact.done) = No such file or directory: 2.889 sec on http probe failed: 2, retry 2024/11/19-15:17:09.340423 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-01 2024/11/19-15:17:09.340481 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00066/att-01/compact.done) = No such file or directory: 2.885 sec on http probe failed: 2, retry 2024/11/19-15:17:09.812649 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-01 2024/11/19-15:17:09.812745 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00061/att-01/compact.done) = No such file or directory: 5.373 sec on http probe failed: 4, retry 2024/11/19-15:17:09.969219 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-00 2024/11/19-15:17:09.969317 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00069/att-00/compact.done) = No such file or directory: 7.202 sec on http probe failed: 3, retry 2024/11/19-15:17:10.026718 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-02 2024/11/19-15:17:10.026815 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00054/att-02/compact.done) = No such file or directory: 9.014 sec on http probe failed: 3, retry 2024/11/19-15:17:10.110301 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-01 2024/11/19-15:17:10.110390 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00068/att-01/compact.done) = No such file or directory: 1.591 sec on http probe failed: 1, retry 2024/11/19-15:17:10.428388 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-01 2024/11/19-15:17:10.428510 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00067/att-01/compact.done) = No such file or directory: 4.095 sec on http probe failed: 3, retry 2024/11/19-15:17:10.610545 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-01 2024/11/19-15:17:10.610636 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00065/att-01/compact.done) = No such file or directory: 4.178 sec on http probe failed: 3, retry 2024/11/19-15:17:10.628517 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-01 2024/11/19-15:17:10.628602 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00066/att-01/compact.done) = No such file or directory: 4.171 sec on http probe failed: 3, retry 2024/11/19-15:17:11.075982 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-01 2024/11/19-15:17:11.076058 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00061/att-01/compact.done) = No such file or directory: 6.640 sec on http probe failed: 5, giveup 2024/11/19-15:17:11.076077 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00061/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 62.907 sec, real 6.926 (submit 281.322 ms wait 6.640 s), input 62.907 MiB, send shutdown 2024/11/19-15:17:11.080625 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-01 2024/11/19-15:17:11.080847 139850978166528 [dcompact_etcd.cc:1773] job-00061/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:11.341331 139850978166528 [dcompact_etcd.cc:1163] job-00061/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.260 sec, input = 62.907 MiB 2024/11/19-15:17:11.389157 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-01 2024/11/19-15:17:11.389232 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00068/att-01/compact.done) = No such file or directory: 2.873 sec on http probe failed: 2, retry 2024/11/19-15:17:11.692793 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-01 2024/11/19-15:17:11.692868 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00067/att-01/compact.done) = No such file or directory: 5.362 sec on http probe failed: 4, retry 2024/11/19-15:17:11.897722 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-01 2024/11/19-15:17:11.897805 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00065/att-01/compact.done) = No such file or directory: 5.463 sec on http probe failed: 4, retry 2024/11/19-15:17:11.917678 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-01 2024/11/19-15:17:11.917755 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00066/att-01/compact.done) = No such file or directory: 5.459 sec on http probe failed: 4, retry 2024/11/19-15:17:11.977516 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-00 2024/11/19-15:17:11.977625 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00069/att-00/compact.done) = No such file or directory: 9.210 sec on http probe failed: 4, retry 2024/11/19-15:17:12.034752 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-02 2024/11/19-15:17:12.034847 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00054/att-02/compact.done) = No such file or directory: 11.020 sec on http probe failed: 4, retry 2024/11/19-15:17:12.671154 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-01 2024/11/19-15:17:12.671249 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00068/att-01/compact.done) = No such file or directory: 4.152 sec on http probe failed: 3, retry 2024/11/19-15:17:12.922758 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-02 2024/11/19-15:17:12.922849 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00061/att-02/compact.done) = No such file or directory: 1.573 sec on http probe failed: 1, retry 2024/11/19-15:17:12.957552 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-01 2024/11/19-15:17:12.957616 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00067/att-01/compact.done) = No such file or directory: 6.626 sec on http probe failed: 5, giveup 2024/11/19-15:17:12.957633 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00067/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 62.824 sec, real 6.935 (submit 300.356 ms wait 6.626 s), input 62.824 MiB, send shutdown 2024/11/19-15:17:12.965606 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-01 2024/11/19-15:17:12.965823 139851158521600 [dcompact_etcd.cc:1773] job-00067/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:13.182505 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-01 2024/11/19-15:17:13.182608 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00065/att-01/compact.done) = No such file or directory: 6.750 sec on http probe failed: 5, giveup 2024/11/19-15:17:13.182627 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00065/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 64.000 sec, real 7.061 (submit 306.106 ms wait 6.750 s), input 64.000 MiB, send shutdown 2024/11/19-15:17:13.190593 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-01 2024/11/19-15:17:13.190807 139851137550080 [dcompact_etcd.cc:1773] job-00065/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:13.205682 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-01 2024/11/19-15:17:13.205733 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00066/att-01/compact.done) = No such file or directory: 6.748 sec on http probe failed: 5, giveup 2024/11/19-15:17:13.205747 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00066/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 64.001 sec, real 7.059 (submit 303.144 ms wait 6.748 s), input 64.001 MiB, send shutdown 2024/11/19-15:17:13.210300 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-01 2024/11/19-15:17:13.210486 139851148035840 [dcompact_etcd.cc:1773] job-00066/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:13.237656 139851158521600 [dcompact_etcd.cc:1163] job-00067/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.272 sec, input = 62.824 MiB 2024/11/19-15:17:13.488513 139851137550080 [dcompact_etcd.cc:1163] job-00065/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.298 sec, input = 64.000 MiB 2024/11/19-15:17:13.499454 139851148035840 [dcompact_etcd.cc:1163] job-00066/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 64.001 MiB 2024/11/19-15:17:13.952798 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-01 2024/11/19-15:17:13.952890 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00068/att-01/compact.done) = No such file or directory: 5.434 sec on http probe failed: 4, retry 2024/11/19-15:17:13.983092 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-00 2024/11/19-15:17:13.983156 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00069/att-00/compact.done) = No such file or directory: 11.219 sec on http probe failed: 5, giveup 2024/11/19-15:17:13.983173 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00069/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.237 sec, real 11.507 (submit 283.400 ms wait 11.219 s), input 127.237 MiB, send shutdown 2024/11/19-15:17:13.988487 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-00 2024/11/19-15:17:13.988699 139851179493120 [dcompact_etcd.cc:1773] job-00069/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:14.043571 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-02 2024/11/19-15:17:14.043632 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00054/att-02/compact.done) = No such file or directory: 13.028 sec on http probe failed: 5, giveup 2024/11/19-15:17:14.043649 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00054/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 244.368 sec, real 13.352 (submit 314.535 ms wait 13.028 s), input 244.368 MiB, send shutdown 2024/11/19-15:17:14.051521 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00054/att-02 2024/11/19-15:17:14.051766 139851169007360 [dcompact_etcd.cc:1773] job-00054/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:14.052051 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00054) = success 2024/11/19-15:17:14.052057 139851169007360 [dcompact_etcd.cc:1773] job-00054/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:14.052075 139851169007360 (Original Log Time 2024/11/19-15:16:34.002042) EVENT_LOG_v1 {"time_micros": 1732000594002036, "job": 54, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4437, 4439, 4438, 4440], "score": 2.41768, "input_data_size": 211946500, "oldest_snapshot_seqno": 112836805} 2024/11/19-15:17:14.052081 139851169007360 [compaction_job.cc:2437] [default] [JOB 54] Compacting 4@2 files to L3, score 2.42, subcompactions 1 : 1 2024/11/19-15:17:14.052089 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 52 Base level 2, inputs: [4437(52MB) 4439(52MB) 4438(52MB) 4440(43MB)] 2024/11/19-15:17:14.166712 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4573. Immutable memtables: 0. 2024/11/19-15:17:14.167198 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 70] Syncing log #4443 2024/11/19-15:17:14.167289 139850967680768 (Original Log Time 2024/11/19-15:17:14.166979) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:17:14.167297 139850967680768 [flush_job.cc:879] [default] [JOB 70] Flushing memtable with next log file: 4573 2024/11/19-15:17:14.167317 139850967680768 EVENT_LOG_v1 {"time_micros": 1732000634167312, "job": 70, "event": "flush_started", "num_memtables": 1, "num_entries": 1301043, "num_deletes": 0, "total_data_size": 476459651, "memory_usage": 537656712, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:17:14.167321 139850967680768 [flush_job.cc:910] [default] [JOB 70] Level-0 flush table #4574: started 2024/11/19-15:17:14.167812 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000006.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004574.sst in ConvertToSST, mem_size = 512.749 M, time = 481.481 us 2024/11/19-15:17:14.186518 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-02 2024/11/19-15:17:14.186659 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00061/att-02/compact.done) = No such file or directory: 2.840 sec on http probe failed: 2, retry 2024/11/19-15:17:14.260364 139851179493120 [dcompact_etcd.cc:1163] job-00069/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.272 sec, input = 127.237 MiB 2024/11/19-15:17:14.352286 139851169007360 [compaction_job.cc:703] job-00054: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:17:14.354254 139851169007360 (Original Log Time 2024/11/19-15:17:14.052120) EVENT_LOG_v1 {"time_micros": 1732000634052101, "job": 54, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4437, 4439, 4438, 4440], "score": 2.41768, "input_data_size": 211946500, "oldest_snapshot_seqno": 112836805} 2024/11/19-15:17:14.354276 139851169007360 (Original Log Time 2024/11/19-15:17:14.353356) [compaction_job.cc:2022] [default] [JOB 54] Compacted 4@2 files to L3 => 0 bytes 2024/11/19-15:17:14.354282 139851169007360 (Original Log Time 2024/11/19-15:17:14.353824) [compaction_job.cc:1224] [default] compacted to: files[0 93 11 44 0 0 0] max score 4.98 (92 files need compaction), MB/sec: 706.0 rd, 0.0 wr, level 3, files in(4, 0) out(0 +0 blob) MB in(202.1, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 1769618, records dropped: 1769618 output_compression: Snappy 2024/11/19-15:17:14.354292 139851169007360 (Original Log Time 2024/11/19-15:17:14.353869) EVENT_LOG_v1 {"time_micros": 1732000634353847, "cf": "default", "job": 54, "event": "compaction_finished", "compaction_time_micros": 300210, "compaction_time_cpu_micros": 265864, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 1769618, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 93, 11, 44, 0, 0, 0]} 2024/11/19-15:17:14.356051 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003708.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.356096 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634356088, "job": 54, "event": "table_file_deletion", "file_number": 3708} 2024/11/19-15:17:14.356592 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003714.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.356619 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634356616, "job": 54, "event": "table_file_deletion", "file_number": 3714} 2024/11/19-15:17:14.357697 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003723.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.357730 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634357725, "job": 54, "event": "table_file_deletion", "file_number": 3723} 2024/11/19-15:17:14.366040 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003728.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.366072 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634366068, "job": 54, "event": "table_file_deletion", "file_number": 3728} 2024/11/19-15:17:14.368457 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003729.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.368483 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634368480, "job": 54, "event": "table_file_deletion", "file_number": 3729} 2024/11/19-15:17:14.416291 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003771.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.416317 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634416314, "job": 54, "event": "table_file_deletion", "file_number": 3771} 2024/11/19-15:17:14.416345 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004574.sst): fsize = 512.751 M, time(ms): open: 0.038, seek: 0.014, finish: 0.035, meta: 0.022, Flush: 0.015, sync: 248.386, close: 0.009, all: 248.519 2024/11/19-15:17:14.416794 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003772.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.416809 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634416807, "job": 54, "event": "table_file_deletion", "file_number": 3772} 2024/11/19-15:17:14.417600 139850967680768 [flush_job.cc:1083] [default] [JOB 70] Flush lasted 250310 microseconds, and 235864 cpu microseconds. 2024/11/19-15:17:14.417687 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003773.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.417705 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634417703, "job": 54, "event": "table_file_deletion", "file_number": 3773} 2024/11/19-15:17:14.417947 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003814.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.417958 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634417957, "job": 54, "event": "table_file_deletion", "file_number": 3814} 2024/11/19-15:17:14.437485 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003815.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.437500 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634437499, "job": 54, "event": "table_file_deletion", "file_number": 3815} 2024/11/19-15:17:14.437629 139850967680768 (Original Log Time 2024/11/19-15:17:14.416390) [flush_job.cc:1029] [default] [JOB 70] Level-0 flush table #4574: 537658138 bytes OK (needs compaction) 2024/11/19-15:17:14.437638 139850967680768 (Original Log Time 2024/11/19-15:17:14.417615) [memtable_list.cc:560] [default] Level-0 commit table #4574 started 2024/11/19-15:17:14.437639 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003816.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.437641 139850967680768 (Original Log Time 2024/11/19-15:17:14.437540) [memtable_list.cc:759] [default] Level-0 commit table #4574: memtable #1 done 2024/11/19-15:17:14.437645 139850967680768 (Original Log Time 2024/11/19-15:17:14.437561) EVENT_LOG_v1 {"time_micros": 1732000634437556, "job": 70, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [1, 93, 11, 44, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:17:14.437648 139850967680768 (Original Log Time 2024/11/19-15:17:14.437601) [db_impl_compaction_flush.cc:310] [default] Level summary: files[1 93 11 44 0 0 0] max score 4.98 (93 files need compaction) 2024/11/19-15:17:14.437649 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634437647, "job": 54, "event": "table_file_deletion", "file_number": 3816} 2024/11/19-15:17:14.437887 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003819.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.437898 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634437897, "job": 54, "event": "table_file_deletion", "file_number": 3819} 2024/11/19-15:17:14.438125 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003821.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.438135 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634438134, "job": 54, "event": "table_file_deletion", "file_number": 3821} 2024/11/19-15:17:14.438399 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003822.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.438409 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634438408, "job": 54, "event": "table_file_deletion", "file_number": 3822} 2024/11/19-15:17:14.439697 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003848.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.439714 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634439712, "job": 54, "event": "table_file_deletion", "file_number": 3848} 2024/11/19-15:17:14.440928 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003905.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.440945 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634440943, "job": 54, "event": "table_file_deletion", "file_number": 3905} 2024/11/19-15:17:14.441307 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003906.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.441318 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634441317, "job": 54, "event": "table_file_deletion", "file_number": 3906} 2024/11/19-15:17:14.441790 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003907.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.441802 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634441800, "job": 54, "event": "table_file_deletion", "file_number": 3907} 2024/11/19-15:17:14.442210 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003910.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.442221 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634442220, "job": 54, "event": "table_file_deletion", "file_number": 3910} 2024/11/19-15:17:14.443114 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003911.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.443130 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634443129, "job": 54, "event": "table_file_deletion", "file_number": 3911} 2024/11/19-15:17:14.445743 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003918.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.445758 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634445756, "job": 54, "event": "table_file_deletion", "file_number": 3918} 2024/11/19-15:17:14.446619 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003939.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.446634 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634446632, "job": 54, "event": "table_file_deletion", "file_number": 3939} 2024/11/19-15:17:14.447428 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003940.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.447442 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634447440, "job": 54, "event": "table_file_deletion", "file_number": 3940} 2024/11/19-15:17:14.448339 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003946.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.448354 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634448353, "job": 54, "event": "table_file_deletion", "file_number": 3946} 2024/11/19-15:17:14.448755 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003947.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.448766 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634448765, "job": 54, "event": "table_file_deletion", "file_number": 3947} 2024/11/19-15:17:14.449417 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003948.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.449428 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634449427, "job": 54, "event": "table_file_deletion", "file_number": 3948} 2024/11/19-15:17:14.450202 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003949.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.450216 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634450215, "job": 54, "event": "table_file_deletion", "file_number": 3949} 2024/11/19-15:17:14.450611 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003950.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.450622 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634450621, "job": 54, "event": "table_file_deletion", "file_number": 3950} 2024/11/19-15:17:14.450759 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003952.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.450767 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634450766, "job": 54, "event": "table_file_deletion", "file_number": 3952} 2024/11/19-15:17:14.451145 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003954.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.451155 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634451154, "job": 54, "event": "table_file_deletion", "file_number": 3954} 2024/11/19-15:17:14.451794 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003955.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.451805 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634451804, "job": 54, "event": "table_file_deletion", "file_number": 3955} 2024/11/19-15:17:14.452560 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003956.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.452576 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634452574, "job": 54, "event": "table_file_deletion", "file_number": 3956} 2024/11/19-15:17:14.452986 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003957.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.452997 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634452996, "job": 54, "event": "table_file_deletion", "file_number": 3957} 2024/11/19-15:17:14.453269 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003958.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.453279 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634453278, "job": 54, "event": "table_file_deletion", "file_number": 3958} 2024/11/19-15:17:14.453548 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003961.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.453558 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634453557, "job": 54, "event": "table_file_deletion", "file_number": 3961} 2024/11/19-15:17:14.453738 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003968.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.453747 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634453746, "job": 54, "event": "table_file_deletion", "file_number": 3968} 2024/11/19-15:17:14.454026 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003969.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.454036 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634454035, "job": 54, "event": "table_file_deletion", "file_number": 3969} 2024/11/19-15:17:14.454274 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003970.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.454284 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634454283, "job": 54, "event": "table_file_deletion", "file_number": 3970} 2024/11/19-15:17:14.454579 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003972.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.454592 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634454590, "job": 54, "event": "table_file_deletion", "file_number": 3972} 2024/11/19-15:17:14.454892 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003974.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.454910 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634454908, "job": 54, "event": "table_file_deletion", "file_number": 3974} 2024/11/19-15:17:14.455212 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004017.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.455228 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634455226, "job": 54, "event": "table_file_deletion", "file_number": 4017} 2024/11/19-15:17:14.456577 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004078.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.456595 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634456593, "job": 54, "event": "table_file_deletion", "file_number": 4078} 2024/11/19-15:17:14.456742 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004081.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.456762 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634456760, "job": 54, "event": "table_file_deletion", "file_number": 4081} 2024/11/19-15:17:14.459065 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004087.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.459085 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634459082, "job": 54, "event": "table_file_deletion", "file_number": 4087} 2024/11/19-15:17:14.461386 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004088.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.461405 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634461403, "job": 54, "event": "table_file_deletion", "file_number": 4088} 2024/11/19-15:17:14.463706 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004090.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.463725 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634463723, "job": 54, "event": "table_file_deletion", "file_number": 4090} 2024/11/19-15:17:14.470607 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004110.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.470627 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634470625, "job": 54, "event": "table_file_deletion", "file_number": 4110} 2024/11/19-15:17:14.474091 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004115.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.474110 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634474108, "job": 54, "event": "table_file_deletion", "file_number": 4115} 2024/11/19-15:17:14.481986 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004122.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.482007 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634482005, "job": 54, "event": "table_file_deletion", "file_number": 4122} 2024/11/19-15:17:14.486083 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004123.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.486102 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634486100, "job": 54, "event": "table_file_deletion", "file_number": 4123} 2024/11/19-15:17:14.493494 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004125.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.493513 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634493511, "job": 54, "event": "table_file_deletion", "file_number": 4125} 2024/11/19-15:17:14.500839 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004126.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.500858 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634500856, "job": 54, "event": "table_file_deletion", "file_number": 4126} 2024/11/19-15:17:14.508286 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004127.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.508306 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634508304, "job": 54, "event": "table_file_deletion", "file_number": 4127} 2024/11/19-15:17:14.516942 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004128.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.516963 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634516961, "job": 54, "event": "table_file_deletion", "file_number": 4128} 2024/11/19-15:17:14.524323 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004129.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.524344 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634524342, "job": 54, "event": "table_file_deletion", "file_number": 4129} 2024/11/19-15:17:14.531610 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004130.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.531630 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634531628, "job": 54, "event": "table_file_deletion", "file_number": 4130} 2024/11/19-15:17:14.538846 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004131.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.538865 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634538863, "job": 54, "event": "table_file_deletion", "file_number": 4131} 2024/11/19-15:17:14.546142 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004132.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.546161 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634546159, "job": 54, "event": "table_file_deletion", "file_number": 4132} 2024/11/19-15:17:14.553782 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004133.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.553802 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634553800, "job": 54, "event": "table_file_deletion", "file_number": 4133} 2024/11/19-15:17:14.561312 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004134.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.561331 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634561329, "job": 54, "event": "table_file_deletion", "file_number": 4134} 2024/11/19-15:17:14.568336 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004136.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.568355 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634568353, "job": 54, "event": "table_file_deletion", "file_number": 4136} 2024/11/19-15:17:14.575270 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004137.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.575289 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634575287, "job": 54, "event": "table_file_deletion", "file_number": 4137} 2024/11/19-15:17:14.582276 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004143.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.582295 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634582293, "job": 54, "event": "table_file_deletion", "file_number": 4143} 2024/11/19-15:17:14.589368 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004144.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.589394 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634589391, "job": 54, "event": "table_file_deletion", "file_number": 4144} 2024/11/19-15:17:14.596188 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004145.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.596207 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634596205, "job": 54, "event": "table_file_deletion", "file_number": 4145} 2024/11/19-15:17:14.602997 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004146.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.603017 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634603015, "job": 54, "event": "table_file_deletion", "file_number": 4146} 2024/11/19-15:17:14.677772 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004325.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.677808 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634677804, "job": 54, "event": "table_file_deletion", "file_number": 4325} 2024/11/19-15:17:14.679466 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004326.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.679485 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634679483, "job": 54, "event": "table_file_deletion", "file_number": 4326} 2024/11/19-15:17:14.680994 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004327.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.681009 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634681007, "job": 54, "event": "table_file_deletion", "file_number": 4327} 2024/11/19-15:17:14.682513 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004328.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.682540 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634682538, "job": 54, "event": "table_file_deletion", "file_number": 4328} 2024/11/19-15:17:14.684044 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004329.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.684058 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634684056, "job": 54, "event": "table_file_deletion", "file_number": 4329} 2024/11/19-15:17:14.685552 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004330.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.685565 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634685563, "job": 54, "event": "table_file_deletion", "file_number": 4330} 2024/11/19-15:17:14.687044 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004332.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.687059 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634687057, "job": 54, "event": "table_file_deletion", "file_number": 4332} 2024/11/19-15:17:14.688672 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004333.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.688687 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634688686, "job": 54, "event": "table_file_deletion", "file_number": 4333} 2024/11/19-15:17:14.691761 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004334.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.691780 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634691777, "job": 54, "event": "table_file_deletion", "file_number": 4334} 2024/11/19-15:17:14.693398 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004335.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.693413 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634693412, "job": 54, "event": "table_file_deletion", "file_number": 4335} 2024/11/19-15:17:14.699663 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004336.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.699682 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634699680, "job": 54, "event": "table_file_deletion", "file_number": 4336} 2024/11/19-15:17:14.706900 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004341.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.707152 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634706917, "job": 54, "event": "table_file_deletion", "file_number": 4341} 2024/11/19-15:17:14.713664 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004342.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.713683 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634713681, "job": 54, "event": "table_file_deletion", "file_number": 4342} 2024/11/19-15:17:14.720378 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004343.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.720397 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634720395, "job": 54, "event": "table_file_deletion", "file_number": 4343} 2024/11/19-15:17:14.727550 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004344.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.727569 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634727567, "job": 54, "event": "table_file_deletion", "file_number": 4344} 2024/11/19-15:17:14.734277 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004345.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.734297 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634734295, "job": 54, "event": "table_file_deletion", "file_number": 4345} 2024/11/19-15:17:14.741021 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004346.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.741040 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634741038, "job": 54, "event": "table_file_deletion", "file_number": 4346} 2024/11/19-15:17:14.747810 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004347.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.747829 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634747827, "job": 54, "event": "table_file_deletion", "file_number": 4347} 2024/11/19-15:17:14.754438 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004348.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.754457 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634754455, "job": 54, "event": "table_file_deletion", "file_number": 4348} 2024/11/19-15:17:14.761113 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004349.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.761131 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634761130, "job": 54, "event": "table_file_deletion", "file_number": 4349} 2024/11/19-15:17:14.768179 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004350.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.768197 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634768196, "job": 54, "event": "table_file_deletion", "file_number": 4350} 2024/11/19-15:17:14.772230 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004359.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.772248 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634772246, "job": 54, "event": "table_file_deletion", "file_number": 4359} 2024/11/19-15:17:14.776127 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004361.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.776145 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634776143, "job": 54, "event": "table_file_deletion", "file_number": 4361} 2024/11/19-15:17:14.780245 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004365.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.780263 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634780261, "job": 54, "event": "table_file_deletion", "file_number": 4365} 2024/11/19-15:17:14.784282 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004366.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.784301 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634784299, "job": 54, "event": "table_file_deletion", "file_number": 4366} 2024/11/19-15:17:14.788265 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004369.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.788283 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634788281, "job": 54, "event": "table_file_deletion", "file_number": 4369} 2024/11/19-15:17:14.792263 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004371.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.792281 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634792279, "job": 54, "event": "table_file_deletion", "file_number": 4371} 2024/11/19-15:17:14.796346 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004372.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.796365 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634796362, "job": 54, "event": "table_file_deletion", "file_number": 4372} 2024/11/19-15:17:14.800411 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004374.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.800430 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634800428, "job": 54, "event": "table_file_deletion", "file_number": 4374} 2024/11/19-15:17:14.804444 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004375.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.804463 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634804461, "job": 54, "event": "table_file_deletion", "file_number": 4375} 2024/11/19-15:17:14.808418 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004376.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.808441 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634808438, "job": 54, "event": "table_file_deletion", "file_number": 4376} 2024/11/19-15:17:14.812944 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004377.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.812965 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634812963, "job": 54, "event": "table_file_deletion", "file_number": 4377} 2024/11/19-15:17:14.817021 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004379.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.817041 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634817039, "job": 54, "event": "table_file_deletion", "file_number": 4379} 2024/11/19-15:17:14.817103 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-02 2024/11/19-15:17:14.817205 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00067/att-02/compact.done) = No such file or directory: 1.571 sec on http probe failed: 1, retry 2024/11/19-15:17:14.820988 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004380.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.821008 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634821006, "job": 54, "event": "table_file_deletion", "file_number": 4380} 2024/11/19-15:17:14.824928 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004381.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.824947 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634824945, "job": 54, "event": "table_file_deletion", "file_number": 4381} 2024/11/19-15:17:14.828872 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004383.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.828890 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634828889, "job": 54, "event": "table_file_deletion", "file_number": 4383} 2024/11/19-15:17:14.832788 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004384.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.832806 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634832804, "job": 54, "event": "table_file_deletion", "file_number": 4384} 2024/11/19-15:17:14.836710 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004385.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.836727 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634836726, "job": 54, "event": "table_file_deletion", "file_number": 4385} 2024/11/19-15:17:14.840622 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004386.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.840640 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634840638, "job": 54, "event": "table_file_deletion", "file_number": 4386} 2024/11/19-15:17:14.844610 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004388.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.844629 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634844627, "job": 54, "event": "table_file_deletion", "file_number": 4388} 2024/11/19-15:17:14.848509 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004389.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.848532 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634848530, "job": 54, "event": "table_file_deletion", "file_number": 4389} 2024/11/19-15:17:14.853583 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004390.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.853601 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634853599, "job": 54, "event": "table_file_deletion", "file_number": 4390} 2024/11/19-15:17:14.857509 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004392.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.857533 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634857531, "job": 54, "event": "table_file_deletion", "file_number": 4392} 2024/11/19-15:17:14.861464 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004393.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.861482 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634861480, "job": 54, "event": "table_file_deletion", "file_number": 4393} 2024/11/19-15:17:14.865527 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004395.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.865545 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634865543, "job": 54, "event": "table_file_deletion", "file_number": 4395} 2024/11/19-15:17:14.869752 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004397.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.869770 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634869768, "job": 54, "event": "table_file_deletion", "file_number": 4397} 2024/11/19-15:17:14.873604 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004399.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.873622 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634873620, "job": 54, "event": "table_file_deletion", "file_number": 4399} 2024/11/19-15:17:14.877535 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004401.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.877553 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634877551, "job": 54, "event": "table_file_deletion", "file_number": 4401} 2024/11/19-15:17:14.881547 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004402.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.881565 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634881563, "job": 54, "event": "table_file_deletion", "file_number": 4402} 2024/11/19-15:17:14.885581 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004403.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.885599 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634885597, "job": 54, "event": "table_file_deletion", "file_number": 4403} 2024/11/19-15:17:14.889460 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004404.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.889477 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634889476, "job": 54, "event": "table_file_deletion", "file_number": 4404} 2024/11/19-15:17:14.893525 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004405.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.893543 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634893541, "job": 54, "event": "table_file_deletion", "file_number": 4405} 2024/11/19-15:17:14.897374 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004406.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.897391 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634897389, "job": 54, "event": "table_file_deletion", "file_number": 4406} 2024/11/19-15:17:14.901453 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004407.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.901471 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634901469, "job": 54, "event": "table_file_deletion", "file_number": 4407} 2024/11/19-15:17:14.905354 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004408.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.905372 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634905370, "job": 54, "event": "table_file_deletion", "file_number": 4408} 2024/11/19-15:17:14.909355 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004411.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.909374 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634909372, "job": 54, "event": "table_file_deletion", "file_number": 4411} 2024/11/19-15:17:14.913369 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004413.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.913387 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634913385, "job": 54, "event": "table_file_deletion", "file_number": 4413} 2024/11/19-15:17:14.917348 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004415.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.917365 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634917364, "job": 54, "event": "table_file_deletion", "file_number": 4415} 2024/11/19-15:17:14.921265 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004417.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.921283 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634921281, "job": 54, "event": "table_file_deletion", "file_number": 4417} 2024/11/19-15:17:14.923971 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004419.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.923988 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634923986, "job": 54, "event": "table_file_deletion", "file_number": 4419} 2024/11/19-15:17:14.924561 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004429.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.924574 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634924572, "job": 54, "event": "table_file_deletion", "file_number": 4429} 2024/11/19-15:17:14.931475 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004441.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.931492 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634931490, "job": 54, "event": "table_file_deletion", "file_number": 4441} 2024/11/19-15:17:14.938546 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004442.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:14.938564 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000634938562, "job": 54, "event": "table_file_deletion", "file_number": 4442} 2024/11/19-15:17:14.939268 139851169007360 [compaction_job.cc:2437] [default] [JOB 71] Compacting 4@1 files to L2, score 4.98, subcompactions 1 : 1 2024/11/19-15:17:14.939280 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 69 Base level 1, inputs: [4486(31MB) 4487(31MB) 4488(31MB) 4490(20MB)] 2024/11/19-15:17:15.094718 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-02 2024/11/19-15:17:15.094799 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00065/att-02/compact.done) = No such file or directory: 1.600 sec on http probe failed: 1, retry 2024/11/19-15:17:15.108423 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-02 2024/11/19-15:17:15.108487 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00066/att-02/compact.done) = No such file or directory: 1.600 sec on http probe failed: 1, retry 2024/11/19-15:17:15.231609 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-01 2024/11/19-15:17:15.231692 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00068/att-01/compact.done) = No such file or directory: 6.716 sec on http probe failed: 5, giveup 2024/11/19-15:17:15.231711 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00068/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 63.642 sec, real 6.992 (submit 270.875 ms wait 6.716 s), input 63.642 MiB, send shutdown 2024/11/19-15:17:15.244088 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-01 2024/11/19-15:17:15.244314 139851127064320 [dcompact_etcd.cc:1773] job-00068/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:15.248129 139851169007360 [dcompact_etcd.cc:1163] job-00071/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.308 sec, input = 115.378 MiB 2024/11/19-15:17:15.454454 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-02 2024/11/19-15:17:15.454532 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00061/att-02/compact.done) = No such file or directory: 4.104 sec on http probe failed: 3, retry 2024/11/19-15:17:15.509233 139851127064320 [dcompact_etcd.cc:1163] job-00068/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.265 sec, input = 63.642 MiB 2024/11/19-15:17:16.082079 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-02 2024/11/19-15:17:16.082158 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00067/att-02/compact.done) = No such file or directory: 2.836 sec on http probe failed: 2, retry 2024/11/19-15:17:16.383603 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-02 2024/11/19-15:17:16.383705 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00065/att-02/compact.done) = No such file or directory: 2.886 sec on http probe failed: 2, retry 2024/11/19-15:17:16.396768 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-02 2024/11/19-15:17:16.396873 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00066/att-02/compact.done) = No such file or directory: 2.889 sec on http probe failed: 2, retry 2024/11/19-15:17:16.718866 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-02 2024/11/19-15:17:16.718949 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00061/att-02/compact.done) = No such file or directory: 5.371 sec on http probe failed: 4, retry 2024/11/19-15:17:17.108331 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-02 2024/11/19-15:17:17.108414 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00068/att-02/compact.done) = No such file or directory: 1.591 sec on http probe failed: 1, retry 2024/11/19-15:17:17.348699 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-02 2024/11/19-15:17:17.348790 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00067/att-02/compact.done) = No such file or directory: 4.101 sec on http probe failed: 3, retry 2024/11/19-15:17:17.450305 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-01 2024/11/19-15:17:17.450427 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00069/att-01/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:17:17.669953 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-02 2024/11/19-15:17:17.670050 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00065/att-02/compact.done) = No such file or directory: 4.175 sec on http probe failed: 3, retry 2024/11/19-15:17:17.685277 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-02 2024/11/19-15:17:17.685337 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00066/att-02/compact.done) = No such file or directory: 4.178 sec on http probe failed: 3, retry 2024/11/19-15:17:17.982595 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-02 2024/11/19-15:17:17.982689 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00061/att-02/compact.done) = No such file or directory: 6.636 sec on http probe failed: 5, giveup 2024/11/19-15:17:17.982714 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00061/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 62.907 sec, real 6.902 (submit 260.409 ms wait 6.636 s), input 62.907 MiB, send shutdown 2024/11/19-15:17:17.988711 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00061/att-02 2024/11/19-15:17:17.988963 139850978166528 [dcompact_etcd.cc:1773] job-00061/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:17.990850 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00061) = success 2024/11/19-15:17:17.990864 139850978166528 [dcompact_etcd.cc:1773] job-00061/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:17:17.990889 139850978166528 (Original Log Time 2024/11/19-15:16:57.177960) EVENT_LOG_v1 {"time_micros": 1732000617177952, "job": 61, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4462], "score": 0.314151, "input_data_size": 65007469, "oldest_snapshot_seqno": -1} 2024/11/19-15:17:17.990900 139850978166528 [compaction_job.cc:2437] [default] [JOB 61] Compacting 1@2 files to L3, score 0.31, subcompactions 1 : 1 2024/11/19-15:17:17.990910 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 59 Base level 2, inputs: [4462(61MB)] 2024/11/19-15:17:18.005104 139850978166528 [compaction_job.cc:703] job-00061: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:17:18.005667 139850978166528 (Original Log Time 2024/11/19-15:17:17.990962) EVENT_LOG_v1 {"time_micros": 1732000637990939, "job": 61, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4462], "score": 0.314151, "input_data_size": 65007469, "oldest_snapshot_seqno": -1} 2024/11/19-15:17:18.005680 139850978166528 (Original Log Time 2024/11/19-15:17:18.005189) [compaction_job.cc:2022] [default] [JOB 61] Compacted 1@2 files to L3 => 0 bytes 2024/11/19-15:17:18.005687 139850978166528 (Original Log Time 2024/11/19-15:17:18.005523) [compaction_job.cc:1224] [default] compacted to: files[1 93 10 44 0 0 0] max score 4.76 (89 files need compaction), MB/sec: 4591.6 rd, 0.0 wr, level 3, files in(1, 0) out(0 +0 blob) MB in(62.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 63611, records dropped: 63611 output_compression: Snappy 2024/11/19-15:17:18.005709 139850978166528 (Original Log Time 2024/11/19-15:17:18.005555) EVENT_LOG_v1 {"time_micros": 1732000638005540, "cf": "default", "job": 61, "event": "compaction_finished", "compaction_time_micros": 14158, "compaction_time_cpu_micros": 14043, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 63611, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 93, 10, 44, 0, 0, 0]} 2024/11/19-15:17:18.006010 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003802.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:18.006035 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000638006031, "job": 61, "event": "table_file_deletion", "file_number": 3802} 2024/11/19-15:17:18.006201 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003953.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:18.006215 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000638006213, "job": 61, "event": "table_file_deletion", "file_number": 3953} 2024/11/19-15:17:18.006373 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004082.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:18.006386 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000638006384, "job": 61, "event": "table_file_deletion", "file_number": 4082} 2024/11/19-15:17:18.012177 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004353.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:18.012204 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000638012201, "job": 61, "event": "table_file_deletion", "file_number": 4353} 2024/11/19-15:17:18.017993 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004357.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:18.018018 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000638018016, "job": 61, "event": "table_file_deletion", "file_number": 4357} 2024/11/19-15:17:18.023303 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004409.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:18.023331 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000638023328, "job": 61, "event": "table_file_deletion", "file_number": 4409} 2024/11/19-15:17:18.029154 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004410.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:18.029181 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000638029178, "job": 61, "event": "table_file_deletion", "file_number": 4410} 2024/11/19-15:17:18.035101 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004412.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:18.035129 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000638035125, "job": 61, "event": "table_file_deletion", "file_number": 4412} 2024/11/19-15:17:18.040908 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004414.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:18.041033 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000638041029, "job": 61, "event": "table_file_deletion", "file_number": 4414} 2024/11/19-15:17:18.046869 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004416.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:18.046895 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000638046892, "job": 61, "event": "table_file_deletion", "file_number": 4416} 2024/11/19-15:17:18.052643 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004418.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:18.052669 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000638052666, "job": 61, "event": "table_file_deletion", "file_number": 4418} 2024/11/19-15:17:18.058545 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004420.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:18.058570 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000638058567, "job": 61, "event": "table_file_deletion", "file_number": 4420} 2024/11/19-15:17:18.064534 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004421.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:18.064559 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000638064556, "job": 61, "event": "table_file_deletion", "file_number": 4421} 2024/11/19-15:17:18.070349 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004422.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:18.070373 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000638070370, "job": 61, "event": "table_file_deletion", "file_number": 4422} 2024/11/19-15:17:18.074623 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004423.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:18.074645 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000638074642, "job": 61, "event": "table_file_deletion", "file_number": 4423} 2024/11/19-15:17:18.074817 139850978166528 [compaction_job.cc:2437] [default] [JOB 72] Compacting 4@1 files to L2, score 4.76, subcompactions 1 : 1 2024/11/19-15:17:18.074826 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 70 Base level 1, inputs: [4491(4758KB) 4472(31MB) 4481(3675KB) 4483(20MB)] 2024/11/19-15:17:18.138392 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-00 2024/11/19-15:17:18.138496 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00071/att-00/compact.done) = No such file or directory: 2.885 sec on http probe failed: 1, retry 2024/11/19-15:17:18.372881 139850978166528 [dcompact_etcd.cc:1163] job-00072/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.298 sec, input = 57.222 MiB 2024/11/19-15:17:18.386393 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-02 2024/11/19-15:17:18.386455 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00068/att-02/compact.done) = No such file or directory: 2.872 sec on http probe failed: 2, retry 2024/11/19-15:17:18.611247 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-02 2024/11/19-15:17:18.611325 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00067/att-02/compact.done) = No such file or directory: 5.368 sec on http probe failed: 4, retry 2024/11/19-15:17:18.955226 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-02 2024/11/19-15:17:18.955299 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00065/att-02/compact.done) = No such file or directory: 5.462 sec on http probe failed: 4, retry 2024/11/19-15:17:18.974422 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-02 2024/11/19-15:17:18.974484 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00066/att-02/compact.done) = No such file or directory: 5.466 sec on http probe failed: 4, retry 2024/11/19-15:17:19.458601 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-01 2024/11/19-15:17:19.458688 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00069/att-01/compact.done) = No such file or directory: 5.190 sec on http probe failed: 2, retry 2024/11/19-15:17:19.664544 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-02 2024/11/19-15:17:19.664639 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00068/att-02/compact.done) = No such file or directory: 4.150 sec on http probe failed: 3, retry 2024/11/19-15:17:19.812147 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-00 2024/11/19-15:17:19.812228 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00072/att-00/compact.done) = No such file or directory: 1.431 sec on http probe failed: 1, retry 2024/11/19-15:17:19.875583 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-02 2024/11/19-15:17:19.875644 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00067/att-02/compact.done) = No such file or directory: 6.630 sec on http probe failed: 5, giveup 2024/11/19-15:17:19.875661 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00067/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 62.824 sec, real 6.910 (submit 271.763 ms wait 6.630 s), input 62.824 MiB, send shutdown 2024/11/19-15:17:19.883257 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00067/att-02 2024/11/19-15:17:19.883495 139851158521600 [dcompact_etcd.cc:1773] job-00067/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:19.885359 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00067) = success 2024/11/19-15:17:19.885367 139851158521600 [dcompact_etcd.cc:1773] job-00067/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:17:19.885385 139851158521600 (Original Log Time 2024/11/19-15:16:59.065710) EVENT_LOG_v1 {"time_micros": 1732000619065705, "job": 67, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4504], "score": 0.314146, "input_data_size": 64922157, "oldest_snapshot_seqno": 113975600} 2024/11/19-15:17:19.885393 139851158521600 [compaction_job.cc:2437] [default] [JOB 67] Compacting 1@2 files to L3, score 0.31, subcompactions 1 : 1 2024/11/19-15:17:19.885400 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 65 Base level 2, inputs: [4504(61MB)] 2024/11/19-15:17:19.892496 139851158521600 [compaction_job.cc:703] job-00067: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:17:19.892864 139851158521600 (Original Log Time 2024/11/19-15:17:19.885434) EVENT_LOG_v1 {"time_micros": 1732000639885412, "job": 67, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4504], "score": 0.314146, "input_data_size": 64922157, "oldest_snapshot_seqno": 113975600} 2024/11/19-15:17:19.892872 139851158521600 (Original Log Time 2024/11/19-15:17:19.892549) [compaction_job.cc:2022] [default] [JOB 67] Compacted 1@2 files to L3 => 0 bytes 2024/11/19-15:17:19.892876 139851158521600 (Original Log Time 2024/11/19-15:17:19.892779) [compaction_job.cc:1224] [default] compacted to: files[1 93 9 44 0 0 0] max score 4.64 (85 files need compaction), MB/sec: 9180.2 rd, 0.0 wr, level 3, files in(1, 0) out(0 +0 blob) MB in(61.9, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 63527, records dropped: 63527 output_compression: Snappy 2024/11/19-15:17:19.892888 139851158521600 (Original Log Time 2024/11/19-15:17:19.892800) EVENT_LOG_v1 {"time_micros": 1732000639892789, "cf": "default", "job": 67, "event": "compaction_finished", "compaction_time_micros": 7072, "compaction_time_cpu_micros": 7025, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 63527, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 93, 9, 44, 0, 0, 0]} 2024/11/19-15:17:19.892975 139851158521600 [compaction_job.cc:2437] [default] [JOB 73] Compacting 4@1 files to L2, score 4.64, subcompactions 1 : 1 2024/11/19-15:17:19.892983 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 71 Base level 1, inputs: [4492(31MB) 4494(31MB) 4496(31MB) 4499(31MB)] 2024/11/19-15:17:20.147468 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-00 2024/11/19-15:17:20.147566 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00071/att-00/compact.done) = No such file or directory: 4.890 sec on http probe failed: 2, retry 2024/11/19-15:17:20.186147 139851158521600 [dcompact_etcd.cc:1163] job-00073/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.293 sec, input = 127.818 MiB 2024/11/19-15:17:20.244140 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-02 2024/11/19-15:17:20.244205 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00065/att-02/compact.done) = No such file or directory: 6.747 sec on http probe failed: 5, giveup 2024/11/19-15:17:20.244221 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00065/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 64.000 sec, real 7.053 (submit 297.633 ms wait 6.747 s), input 64.000 MiB, send shutdown 2024/11/19-15:17:20.252765 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00065/att-02 2024/11/19-15:17:20.252983 139851137550080 [dcompact_etcd.cc:1773] job-00065/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:20.254968 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00065) = success 2024/11/19-15:17:20.254976 139851137550080 [dcompact_etcd.cc:1773] job-00065/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:17:20.254995 139851137550080 (Original Log Time 2024/11/19-15:16:59.062002) EVENT_LOG_v1 {"time_micros": 1732000619061997, "job": 65, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4457], "score": 0.314146, "input_data_size": 66137133, "oldest_snapshot_seqno": 113975600} 2024/11/19-15:17:20.255003 139851137550080 [compaction_job.cc:2437] [default] [JOB 65] Compacting 1@2 files to L3, score 0.31, subcompactions 1 : 1 2024/11/19-15:17:20.255010 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 63 Base level 2, inputs: [4457(63MB)] 2024/11/19-15:17:20.262122 139851137550080 [compaction_job.cc:703] job-00065: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:17:20.262499 139851137550080 (Original Log Time 2024/11/19-15:17:20.255039) EVENT_LOG_v1 {"time_micros": 1732000640255023, "job": 65, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4457], "score": 0.314146, "input_data_size": 66137133, "oldest_snapshot_seqno": 113975600} 2024/11/19-15:17:20.262508 139851137550080 (Original Log Time 2024/11/19-15:17:20.262180) [compaction_job.cc:2022] [default] [JOB 65] Compacted 1@2 files to L3 => 0 bytes 2024/11/19-15:17:20.262511 139851137550080 (Original Log Time 2024/11/19-15:17:20.262414) [compaction_job.cc:1224] [default] compacted to: files[1 93 8 44 0 0 0] max score 4.39 (81 files need compaction), MB/sec: 9324.3 rd, 0.0 wr, level 3, files in(1, 0) out(0 +0 blob) MB in(63.1, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 64714, records dropped: 64714 output_compression: Snappy 2024/11/19-15:17:20.262525 139851137550080 (Original Log Time 2024/11/19-15:17:20.262436) EVENT_LOG_v1 {"time_micros": 1732000640262425, "cf": "default", "job": 65, "event": "compaction_finished", "compaction_time_micros": 7093, "compaction_time_cpu_micros": 7051, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 64714, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 93, 8, 44, 0, 0, 0]} 2024/11/19-15:17:20.262685 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004445.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.262701 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000640262698, "job": 65, "event": "table_file_deletion", "file_number": 4445} 2024/11/19-15:17:20.262811 139851137550080 [compaction_job.cc:2437] [default] [JOB 74] Compacting 4@1 files to L2, score 4.39, subcompactions 1 : 1 2024/11/19-15:17:20.262820 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 72 Base level 1, inputs: [4503(31MB) 4506(18MB) 4474(31MB) 4479(31MB)] 2024/11/19-15:17:20.263765 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-02 2024/11/19-15:17:20.263816 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00066/att-02/compact.done) = No such file or directory: 6.755 sec on http probe failed: 5, giveup 2024/11/19-15:17:20.263830 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00066/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 64.001 sec, real 7.053 (submit 288.906 ms wait 6.755 s), input 64.001 MiB, send shutdown 2024/11/19-15:17:20.270510 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00066/att-02 2024/11/19-15:17:20.270693 139851148035840 [dcompact_etcd.cc:1773] job-00066/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:20.272432 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00066) = success 2024/11/19-15:17:20.272438 139851148035840 [dcompact_etcd.cc:1773] job-00066/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:17:20.272453 139851148035840 (Original Log Time 2024/11/19-15:16:59.064503) EVENT_LOG_v1 {"time_micros": 1732000619064498, "job": 66, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4498], "score": 0.314146, "input_data_size": 66027123, "oldest_snapshot_seqno": 113975600} 2024/11/19-15:17:20.272459 139851148035840 [compaction_job.cc:2437] [default] [JOB 66] Compacting 1@2 files to L3, score 0.31, subcompactions 1 : 1 2024/11/19-15:17:20.272465 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 64 Base level 2, inputs: [4498(62MB)] 2024/11/19-15:17:20.277978 139851148035840 [compaction_job.cc:703] job-00066: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:17:20.278327 139851148035840 (Original Log Time 2024/11/19-15:17:20.272487) EVENT_LOG_v1 {"time_micros": 1732000640272474, "job": 66, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4498], "score": 0.314146, "input_data_size": 66027123, "oldest_snapshot_seqno": 113975600} 2024/11/19-15:17:20.278335 139851148035840 (Original Log Time 2024/11/19-15:17:20.278025) [compaction_job.cc:2022] [default] [JOB 66] Compacted 1@2 files to L3 => 0 bytes 2024/11/19-15:17:20.278338 139851148035840 (Original Log Time 2024/11/19-15:17:20.278227) [compaction_job.cc:1224] [default] compacted to: files[1 93 7 44 0 0 0] max score 4.17 (77 files need compaction), MB/sec: 12002.7 rd, 0.0 wr, level 3, files in(1, 0) out(0 +0 blob) MB in(63.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 64714, records dropped: 64714 output_compression: Snappy 2024/11/19-15:17:20.278343 139851148035840 (Original Log Time 2024/11/19-15:17:20.278244) EVENT_LOG_v1 {"time_micros": 1732000640278235, "cf": "default", "job": 66, "event": "compaction_finished", "compaction_time_micros": 5501, "compaction_time_cpu_micros": 5467, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 64714, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 93, 7, 44, 0, 0, 0]} 2024/11/19-15:17:20.345544 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004340.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.345585 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640345580, "job": 66, "event": "table_file_deletion", "file_number": 4340} 2024/11/19-15:17:20.423111 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004428.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.423156 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640423152, "job": 66, "event": "table_file_deletion", "file_number": 4428} 2024/11/19-15:17:20.488902 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004432.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.488952 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640488948, "job": 66, "event": "table_file_deletion", "file_number": 4432} 2024/11/19-15:17:20.531640 139851137550080 [dcompact_etcd.cc:1163] job-00074/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.268 sec, input = 113.940 MiB 2024/11/19-15:17:20.559606 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004436.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.559645 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640559641, "job": 66, "event": "table_file_deletion", "file_number": 4436} 2024/11/19-15:17:20.631179 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004444.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.631216 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640631212, "job": 66, "event": "table_file_deletion", "file_number": 4444} 2024/11/19-15:17:20.631524 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004446.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.631538 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640631536, "job": 66, "event": "table_file_deletion", "file_number": 4446} 2024/11/19-15:17:20.635621 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004447.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.635644 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640635641, "job": 66, "event": "table_file_deletion", "file_number": 4447} 2024/11/19-15:17:20.635778 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004448.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.635787 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640635786, "job": 66, "event": "table_file_deletion", "file_number": 4448} 2024/11/19-15:17:20.635907 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004449.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.635915 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640635914, "job": 66, "event": "table_file_deletion", "file_number": 4449} 2024/11/19-15:17:20.636103 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004450.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.636113 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640636112, "job": 66, "event": "table_file_deletion", "file_number": 4450} 2024/11/19-15:17:20.640151 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004451.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.640170 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640640168, "job": 66, "event": "table_file_deletion", "file_number": 4451} 2024/11/19-15:17:20.641108 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004452.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.641125 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640641123, "job": 66, "event": "table_file_deletion", "file_number": 4452} 2024/11/19-15:17:20.641818 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004453.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.641831 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640641830, "job": 66, "event": "table_file_deletion", "file_number": 4453} 2024/11/19-15:17:20.642100 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004454.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.642112 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640642110, "job": 66, "event": "table_file_deletion", "file_number": 4454} 2024/11/19-15:17:20.642460 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004455.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.642472 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640642470, "job": 66, "event": "table_file_deletion", "file_number": 4455} 2024/11/19-15:17:20.646525 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004456.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.646544 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640646542, "job": 66, "event": "table_file_deletion", "file_number": 4456} 2024/11/19-15:17:20.648918 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004458.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.649072 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640649070, "job": 66, "event": "table_file_deletion", "file_number": 4458} 2024/11/19-15:17:20.650213 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004459.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.650229 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640650227, "job": 66, "event": "table_file_deletion", "file_number": 4459} 2024/11/19-15:17:20.654247 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004460.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.654265 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640654263, "job": 66, "event": "table_file_deletion", "file_number": 4460} 2024/11/19-15:17:20.658207 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004461.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.658225 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640658223, "job": 66, "event": "table_file_deletion", "file_number": 4461} 2024/11/19-15:17:20.662271 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004463.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.662293 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640662291, "job": 66, "event": "table_file_deletion", "file_number": 4463} 2024/11/19-15:17:20.666486 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004464.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.666507 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640666505, "job": 66, "event": "table_file_deletion", "file_number": 4464} 2024/11/19-15:17:20.670656 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004465.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.670677 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640670675, "job": 66, "event": "table_file_deletion", "file_number": 4465} 2024/11/19-15:17:20.674825 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004466.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.674843 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640674841, "job": 66, "event": "table_file_deletion", "file_number": 4466} 2024/11/19-15:17:20.678784 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004467.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.678801 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640678800, "job": 66, "event": "table_file_deletion", "file_number": 4467} 2024/11/19-15:17:20.682974 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004468.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.682992 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640682990, "job": 66, "event": "table_file_deletion", "file_number": 4468} 2024/11/19-15:17:20.688164 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004469.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.688182 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640688180, "job": 66, "event": "table_file_deletion", "file_number": 4469} 2024/11/19-15:17:20.693161 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004470.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.693179 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640693177, "job": 66, "event": "table_file_deletion", "file_number": 4470} 2024/11/19-15:17:20.694743 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004471.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:20.694760 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000640694758, "job": 66, "event": "table_file_deletion", "file_number": 4471} 2024/11/19-15:17:20.694973 139851148035840 (Original Log Time 2024/11/19-15:17:20.694916) [db_impl_compaction_flush.cc:3374] [default] Manual compaction from level-2 to level-3 from '00000135' seq:72057594037927935, type:22 .. '00000136' seq:0, type:0; will stop at (end) 2024/11/19-15:17:20.694980 139851148035840 [compaction_job.cc:2437] [default] [JOB 75] Compacting 4@2 files to L3, score -1.00, subcompactions 1 : 1 2024/11/19-15:17:20.694987 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 73 Base level 2, inputs: [4501(63MB) 4507(61MB) 4502(63MB) 4508(56MB)] 2024/11/19-15:17:20.942987 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-02 2024/11/19-15:17:20.943087 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00068/att-02/compact.done) = No such file or directory: 5.428 sec on http probe failed: 4, retry 2024/11/19-15:17:20.965628 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-00 2024/11/19-15:17:20.965706 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00072/att-00/compact.done) = No such file or directory: 2.584 sec on http probe failed: 2, retry 2024/11/19-15:17:21.003955 139851148035840 [dcompact_etcd.cc:1163] job-00075/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.308 sec, input = 248.724 MiB 2024/11/19-15:17:21.466776 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-01 2024/11/19-15:17:21.466856 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00069/att-01/compact.done) = No such file or directory: 7.198 sec on http probe failed: 3, retry 2024/11/19-15:17:22.119281 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-00 2024/11/19-15:17:22.119385 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00072/att-00/compact.done) = No such file or directory: 3.737 sec on http probe failed: 3, retry 2024/11/19-15:17:22.153737 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-00 2024/11/19-15:17:22.153804 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00071/att-00/compact.done) = No such file or directory: 6.900 sec on http probe failed: 3, retry 2024/11/19-15:17:22.224352 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-02 2024/11/19-15:17:22.224422 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00068/att-02/compact.done) = No such file or directory: 6.707 sec on http probe failed: 5, giveup 2024/11/19-15:17:22.224440 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00068/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 63.642 sec, real 6.980 (submit 264.834 ms wait 6.707 s), input 63.642 MiB, send shutdown 2024/11/19-15:17:22.232706 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00068/att-02 2024/11/19-15:17:22.232932 139851127064320 [dcompact_etcd.cc:1773] job-00068/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:22.234917 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00068) = success 2024/11/19-15:17:22.234938 139851127064320 [dcompact_etcd.cc:1773] job-00068/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:17:22.234960 139851127064320 (Original Log Time 2024/11/19-15:17:01.169793) EVENT_LOG_v1 {"time_micros": 1732000621169785, "job": 68, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4473, 4475, 4476, 4477], "score": 5.35852, "input_data_size": 67047283, "oldest_snapshot_seqno": 114044668} 2024/11/19-15:17:22.234967 139851127064320 [compaction_job.cc:2437] [default] [JOB 68] Compacting 4@1 files to L2, score 5.36, subcompactions 1 : 1 2024/11/19-15:17:22.234976 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 66 Base level 1, inputs: [4473(5573B) 4475(27KB) 4476(31MB) 4477(31MB)] 2024/11/19-15:17:22.235200 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004575.sst) /storage/mytopling/..//tmp/Topling-82038-kXYmHz 2024/11/19-15:17:22.235298 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004575.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:17:22.235368 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004575.sst): index pass time = 0.00's, 0.992'MB/sec index type = UintIndex_AllOne usrkeys = 14 min-keylen = 4 max-keylen = 4 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 11.43 2024/11/19-15:17:22.235444 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004575.sst): first pass time = 0.00's, 7.817'MB/sec 2024/11/19-15:17:22.235470 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004575.sst): old prealloc_size = 73752011, real_size = 2352 2024/11/19-15:17:22.235484 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004575.sst): index type = UintIndex_AllOne , store type = MixedLenBlobStore usrkeys = 14 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 11.43 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 115.00 avg-zval = 141.71 2024/11/19-15:17:22.235626 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004575.sst): second pass time = 0.00's,10805.369'MB/sec, value only(87.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 22.941'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 11.471'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.364'MB/sec write SST data time = 0.00's, 27.925'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 10805.369'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 14 avg-key = 16.00 avg-zkey = 13.71 avg-val = 115.00 avg-zval = 141.71 usrkeys = 14 avg-key = 8.00 avg-zkey = 13.71 avg-val = 115.00 avg-zval = 141.71 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 3 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.5833 value = 0.8115 dict = 0.00 all = 0.5629 } Zip/UnZip{ index = 1.7143 value = 1.2323 dict = 0.00 all = 1.7764 } ---------------------------- total value len = 2.455846 GB avg = 0.430 KB (by entry num) total key len = 0.141617 GB avg = 0.025 KB total ukey len = 0.095883 GB avg = 0.017 KB total ukey num = 0.005716686 Billion total entry num = 0.005716686 Billion write speed all = 17.609321233 MB/sec (with seq num) write speed all = 17.299274178 MB/sec (without seq num) 2024/11/19-15:17:22.235798 139851127064320 [compaction_job.cc:1948] [default] [JOB 68] Generated table #4575: 14 keys, 3258 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:22.235831 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000642235811, "cf_name": "default", "job": 68, "event": "table_file_creation", "file_number": 4575, "file_size": 3258, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 1984, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 224, "raw_average_key_size": 16, "raw_value_size": 1610, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 14, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-MixedLenBlobStore", "creation_time": 1732000642, "oldest_key_time": 0, "file_creation_time": 1732000642, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4575, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:17:22.235985 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004576.sst) /storage/mytopling/..//tmp/Topling-82038-LtW4LS 2024/11/19-15:17:22.236119 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004576.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:17:22.236241 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004576.sst): first pass time = 0.00's, 90.498'MB/sec 2024/11/19-15:17:22.236273 139834035275520 [fixed_len_key_index.cc:363] FixedLenKeyIndex build_cache Finish: num_keys = 160, fixlen = 8, pref_len = 4, cplen = 3, time = 0.000 sec 2024/11/19-15:17:22.236330 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004576.sst): index pass time = 0.00's, 6.655'MB/sec index type = FixedLenKeyIndex usrkeys = 160 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 59.20 2024/11/19-15:17:22.236376 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004576.sst): old prealloc_size = 73752011, real_size = 29488 2024/11/19-15:17:22.236394 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004576.sst): index type = FixedLenKeyIndex , store type = PlainBlobStore usrkeys = 160 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 59.20 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 120.53 avg-zval = 123.80 2024/11/19-15:17:22.236533 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004576.sst): second pass time = 0.00's,419217.391'MB/sec, value only(85.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 197.471'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 594.975'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 3.407'MB/sec write SST data time = 0.00's, 238.826'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 419217.391'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 160 avg-key = 20.00 avg-zkey = 59.20 avg-val = 120.53 avg-zval = 123.80 usrkeys = 160 avg-key = 12.00 avg-zkey = 59.20 avg-val = 120.53 avg-zval = 123.80 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 5 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.2027 value = 0.9735 dict = 0.00 all = 0.7395 } Zip/UnZip{ index = 4.9333 value = 1.0272 dict = 0.00 all = 1.3523 } ---------------------------- total value len = 2.455865 GB avg = 0.430 KB (by entry num) total key len = 0.141620 GB avg = 0.025 KB total ukey len = 0.095885 GB avg = 0.017 KB total ukey num = 0.005716846 Billion total entry num = 0.005716846 Billion write speed all = 17.609365273 MB/sec (with seq num) write speed all = 17.299311448 MB/sec (without seq num) 2024/11/19-15:17:22.236945 139851127064320 [compaction_job.cc:1948] [default] [JOB 68] Generated table #4576: 160 keys, 30404 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:22.236973 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000642236957, "cf_name": "default", "job": 68, "event": "table_file_creation", "file_number": 4576, "file_size": 30404, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 19808, "index_size": 9472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3200, "raw_average_key_size": 20, "raw_value_size": 19284, "raw_average_value_size": 120, "num_data_blocks": 1, "num_entries": 160, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-FixedLenKeyIndex-PlainBlobStore", "creation_time": 1732000642, "oldest_key_time": 0, "file_creation_time": 1732000642, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4576, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:17:22.237123 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004577.sst) /storage/mytopling/..//tmp/Topling-82038-xFJolV 2024/11/19-15:17:22.297809 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004577.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:17:22.306102 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004577.sst): index pass time = 0.01's, 144.983'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 99735 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:17:22.369762 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004577.sst): first pass time = 0.13's, 502.976'MB/sec 2024/11/19-15:17:22.369815 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004577.sst): old prealloc_size = 73752011, real_size = 64833792 2024/11/19-15:17:22.369887 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004577.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 99735 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0643 GB zip-val = 0.0646 GB avg-val = 644.86 avg-zval = 648.11 2024/11/19-15:17:22.369938 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004577.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.072 sec, Key+Value = 0.067 GB 2024/11/19-15:17:22.438540 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004577.sst): second pass time = 0.00's,251231148.438'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 943.313'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2450.200'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.562'MB/sec write SST data time = 0.02's, 2674.097'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.064'GB zip my value throughput = 251231148.437'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 99735 avg-key = 24.00 avg-zkey = 1.70 avg-val = 644.86 avg-zval = 648.11 usrkeys = 99735 avg-key = 16.00 avg-zkey = 1.70 avg-val = 644.86 avg-zval = 648.11 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7895 UnZipSize{ index = 0.0016 GB value = 0.0643 GB dict = 0.00 MB all = 0.0667 GB } __ZipSize{ index = 0.0002 GB value = 0.0646 GB dict = 0.00 MB all = 0.0648 GB } UnZip/Zip{ index = 9.4339 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9715 } ---------------------------- total value len = 2.520180 GB avg = 0.433 KB (by entry num) total key len = 0.144014 GB avg = 0.025 KB total ukey len = 0.097481 GB avg = 0.017 KB total ukey num = 0.005816581 Billion total entry num = 0.005816581 Billion write speed all = 18.036910975 MB/sec (with seq num) write speed all = 17.721879409 MB/sec (without seq num) 2024/11/19-15:17:22.442836 139851127064320 [compaction_job.cc:1948] [default] [JOB 68] Generated table #4577: 99735 keys, 64810765 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:22.442888 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000642442865, "cf_name": "default", "job": 68, "event": "table_file_creation", "file_number": 4577, "file_size": 64810765, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64639536, "index_size": 169152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2393640, "raw_average_key_size": 24, "raw_value_size": 64315174, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 99735, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000642, "oldest_key_time": 0, "file_creation_time": 1732000642, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4577, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:17:22.458577 139851127064320 [compaction_job.cc:703] job-00068: subcompact[0], size: 0.064844 G, files: /storage/mytopling/.rocksdb [004575,004576,004577] 2024/11/19-15:17:22.459448 139851127064320 (Original Log Time 2024/11/19-15:17:22.235007) EVENT_LOG_v1 {"time_micros": 1732000642234989, "job": 68, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4473, 4475, 4476, 4477], "score": 5.35852, "input_data_size": 67047283, "oldest_snapshot_seqno": 114044668} 2024/11/19-15:17:22.459457 139851127064320 (Original Log Time 2024/11/19-15:17:22.459083) [compaction_job.cc:2022] [default] [JOB 68] Compacted 4@1 files to L2 => 64844427 bytes 2024/11/19-15:17:22.459461 139851127064320 (Original Log Time 2024/11/19-15:17:22.459326) [compaction_job.cc:1224] [default] compacted to: files[1 89 10 44 0 0 0] max score 4.17 (76 files need compaction), MB/sec: 299.9 rd, 290.0 wr, level 2, files in(4, 0) out(3 +0 blob) MB in(63.9, 0.0 +0.0 blob) out(61.8 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 99909, records dropped: 0 output_compression: Snappy 2024/11/19-15:17:22.459473 139851127064320 (Original Log Time 2024/11/19-15:17:22.459352) EVENT_LOG_v1 {"time_micros": 1732000642459339, "cf": "default", "job": 68, "event": "compaction_finished", "compaction_time_micros": 223583, "compaction_time_cpu_micros": 222192, "output_level": 2, "num_output_files": 3, "total_output_size": 64844427, "num_input_records": 99909, "num_output_records": 99909, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 89, 10, 44, 0, 0, 0]} 2024/11/19-15:17:22.466330 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004433.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:22.466356 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000642466353, "job": 68, "event": "table_file_deletion", "file_number": 4433} 2024/11/19-15:17:22.471939 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004434.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:22.471962 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000642471959, "job": 68, "event": "table_file_deletion", "file_number": 4434} 2024/11/19-15:17:22.472090 139851127064320 [compaction_job.cc:2437] [default] [JOB 76] Compacting 4@1 files to L2, score 4.17, subcompactions 1 : 1 2024/11/19-15:17:22.472099 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 74 Base level 1, inputs: [4485(31MB) 4489(31MB) 4493(31MB) 4495(31MB)] 2024/11/19-15:17:22.743911 139851127064320 [dcompact_etcd.cc:1163] job-00076/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.271 sec, input = 127.821 MiB 2024/11/19-15:17:23.272316 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-00 2024/11/19-15:17:23.272416 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00072/att-00/compact.done) = No such file or directory: 4.891 sec on http probe failed: 4, retry 2024/11/19-15:17:23.389651 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-00 2024/11/19-15:17:23.389734 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00074/att-00/compact.done) = No such file or directory: 2.849 sec on http probe failed: 1, retry 2024/11/19-15:17:23.390403 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-00 2024/11/19-15:17:23.390470 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00073/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:17:23.472104 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-01 2024/11/19-15:17:23.472187 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00069/att-01/compact.done) = No such file or directory: 9.207 sec on http probe failed: 4, retry 2024/11/19-15:17:24.158753 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-00 2024/11/19-15:17:24.158844 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00071/att-00/compact.done) = No such file or directory: 8.906 sec on http probe failed: 4, retry 2024/11/19-15:17:24.426521 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-00 2024/11/19-15:17:24.426601 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00072/att-00/compact.done) = No such file or directory: 6.044 sec on http probe failed: 5, giveup 2024/11/19-15:17:24.426619 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00072/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 57.222 sec, real 6.351 (submit 297.590 ms wait 6.044 s), input 57.222 MiB, send shutdown 2024/11/19-15:17:24.434518 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-00 2024/11/19-15:17:24.434743 139850978166528 [dcompact_etcd.cc:1773] job-00072/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:24.710839 139850978166528 [dcompact_etcd.cc:1163] job-00072/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.276 sec, input = 57.222 MiB 2024/11/19-15:17:25.396873 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-00 2024/11/19-15:17:25.396958 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00073/att-00/compact.done) = No such file or directory: 5.204 sec on http probe failed: 2, retry 2024/11/19-15:17:25.399425 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-00 2024/11/19-15:17:25.399484 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00074/att-00/compact.done) = No such file or directory: 4.858 sec on http probe failed: 2, retry 2024/11/19-15:17:25.481834 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-01 2024/11/19-15:17:25.481912 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00069/att-01/compact.done) = No such file or directory: 11.212 sec on http probe failed: 5, giveup 2024/11/19-15:17:25.481940 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00069/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.237 sec, real 11.493 (submit 271.586 ms wait 11.212 s), input 127.237 MiB, send shutdown 2024/11/19-15:17:25.491135 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-01 2024/11/19-15:17:25.491405 139851179493120 [dcompact_etcd.cc:1773] job-00069/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:25.788093 139851179493120 [dcompact_etcd.cc:1163] job-00069/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.297 sec, input = 127.237 MiB 2024/11/19-15:17:25.945231 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-00 2024/11/19-15:17:25.945326 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00076/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:17:26.013103 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-00 2024/11/19-15:17:26.013186 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00075/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:17:26.149771 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-01 2024/11/19-15:17:26.149862 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00072/att-01/compact.done) = No such file or directory: 1.431 sec on http probe failed: 1, retry 2024/11/19-15:17:26.168154 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-00 2024/11/19-15:17:26.168224 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00071/att-00/compact.done) = No such file or directory: 10.911 sec on http probe failed: 5, giveup 2024/11/19-15:17:26.168241 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00071/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 115.378 sec, real 11.228 (submit 308.386 ms wait 10.911 s), input 115.378 MiB, send shutdown 2024/11/19-15:17:26.173548 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-00 2024/11/19-15:17:26.173769 139851169007360 [dcompact_etcd.cc:1773] job-00071/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:26.445320 139851169007360 [dcompact_etcd.cc:1163] job-00071/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.271 sec, input = 115.378 MiB 2024/11/19-15:17:27.302752 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-01 2024/11/19-15:17:27.302839 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00072/att-01/compact.done) = No such file or directory: 2.584 sec on http probe failed: 2, retry 2024/11/19-15:17:27.403373 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-00 2024/11/19-15:17:27.403441 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00073/att-00/compact.done) = No such file or directory: 7.211 sec on http probe failed: 3, retry 2024/11/19-15:17:27.405314 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-00 2024/11/19-15:17:27.405373 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00074/att-00/compact.done) = No such file or directory: 6.868 sec on http probe failed: 3, retry 2024/11/19-15:17:27.950506 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-00 2024/11/19-15:17:27.950580 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00076/att-00/compact.done) = No such file or directory: 5.201 sec on http probe failed: 2, retry 2024/11/19-15:17:28.019508 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-00 2024/11/19-15:17:28.019588 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00075/att-00/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:17:28.452909 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-01 2024/11/19-15:17:28.453013 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00072/att-01/compact.done) = No such file or directory: 3.737 sec on http probe failed: 3, retry 2024/11/19-15:17:28.975396 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-02 2024/11/19-15:17:28.975481 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00069/att-02/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:17:29.335417 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-01 2024/11/19-15:17:29.335510 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00071/att-01/compact.done) = No such file or directory: 2.885 sec on http probe failed: 1, retry 2024/11/19-15:17:29.410811 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-00 2024/11/19-15:17:29.410900 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00074/att-00/compact.done) = No such file or directory: 8.874 sec on http probe failed: 4, retry 2024/11/19-15:17:29.411911 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-00 2024/11/19-15:17:29.411982 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00073/att-00/compact.done) = No such file or directory: 9.217 sec on http probe failed: 4, retry 2024/11/19-15:17:29.605780 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-01 2024/11/19-15:17:29.605866 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00072/att-01/compact.done) = No such file or directory: 4.887 sec on http probe failed: 4, retry 2024/11/19-15:17:29.955605 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-00 2024/11/19-15:17:29.955699 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00076/att-00/compact.done) = No such file or directory: 7.207 sec on http probe failed: 3, retry 2024/11/19-15:17:30.027708 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-00 2024/11/19-15:17:30.027801 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00075/att-00/compact.done) = No such file or directory: 9.016 sec on http probe failed: 3, retry 2024/11/19-15:17:30.758335 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-01 2024/11/19-15:17:30.758435 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00072/att-01/compact.done) = No such file or directory: 6.040 sec on http probe failed: 5, giveup 2024/11/19-15:17:30.758453 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00072/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 57.222 sec, real 6.324 (submit 276.015 ms wait 6.040 s), input 57.222 MiB, send shutdown 2024/11/19-15:17:30.766329 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-01 2024/11/19-15:17:30.766560 139850978166528 [dcompact_etcd.cc:1773] job-00072/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:30.981287 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-02 2024/11/19-15:17:30.981379 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00069/att-02/compact.done) = No such file or directory: 5.187 sec on http probe failed: 2, retry 2024/11/19-15:17:31.066680 139850978166528 [dcompact_etcd.cc:1163] job-00072/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.300 sec, input = 57.222 MiB 2024/11/19-15:17:31.344164 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-01 2024/11/19-15:17:31.344269 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00071/att-01/compact.done) = No such file or directory: 4.890 sec on http probe failed: 2, retry 2024/11/19-15:17:31.419579 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-00 2024/11/19-15:17:31.419645 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00074/att-00/compact.done) = No such file or directory: 10.879 sec on http probe failed: 5, giveup 2024/11/19-15:17:31.419665 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00074/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 113.940 sec, real 11.156 (submit 268.401 ms wait 10.879 s), input 113.940 MiB, send shutdown 2024/11/19-15:17:31.419933 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-00 2024/11/19-15:17:31.419984 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00073/att-00/compact.done) = No such file or directory: 11.226 sec on http probe failed: 5, giveup 2024/11/19-15:17:31.419999 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00073/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.818 sec, real 11.527 (submit 292.689 ms wait 11.226 s), input 127.818 MiB, send shutdown 2024/11/19-15:17:31.425440 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-00 2024/11/19-15:17:31.425650 139851158521600 [dcompact_etcd.cc:1773] job-00073/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:31.427910 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-00 2024/11/19-15:17:31.428155 139851137550080 [dcompact_etcd.cc:1773] job-00074/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:31.681329 139851137550080 [dcompact_etcd.cc:1163] job-00074/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.253 sec, input = 113.940 MiB 2024/11/19-15:17:31.694878 139851158521600 [dcompact_etcd.cc:1163] job-00073/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.269 sec, input = 127.818 MiB 2024/11/19-15:17:31.963978 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-00 2024/11/19-15:17:31.964068 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00076/att-00/compact.done) = No such file or directory: 9.212 sec on http probe failed: 4, retry 2024/11/19-15:17:32.036332 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-00 2024/11/19-15:17:32.036412 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00075/att-00/compact.done) = No such file or directory: 11.024 sec on http probe failed: 4, retry 2024/11/19-15:17:32.414021 139852349699840 [db_impl_write.cc:2181] [default] New memtable created with log file: #4578. Immutable memtables: 0. 2024/11/19-15:17:32.414325 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 77] Syncing log #4573 2024/11/19-15:17:32.414419 139850957195008 (Original Log Time 2024/11/19-15:17:32.414126) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:17:32.414427 139850957195008 [flush_job.cc:879] [default] [JOB 77] Flushing memtable with next log file: 4578 2024/11/19-15:17:32.414446 139850957195008 EVENT_LOG_v1 {"time_micros": 1732000652414441, "job": 77, "event": "flush_started", "num_memtables": 1, "num_entries": 1573678, "num_deletes": 0, "total_data_size": 466275298, "memory_usage": 541064584, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:17:32.414450 139850957195008 [flush_job.cc:910] [default] [JOB 77] Level-0 flush table #4579: started 2024/11/19-15:17:32.414786 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000007.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004579.sst in ConvertToSST, mem_size = 515.999 M, time = 328.152 us 2024/11/19-15:17:32.507075 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-02 2024/11/19-15:17:32.507235 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00072/att-02/compact.done) = No such file or directory: 1.431 sec on http probe failed: 1, retry 2024/11/19-15:17:32.933406 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004579.sst): fsize = 516.001 M, time(ms): open: 0.033, seek: 0.013, finish: 0.036, meta: 0.011, Flush: 0.012, sync: 518.490, close: 0.011, all: 518.606 2024/11/19-15:17:32.934557 139850957195008 [flush_job.cc:1083] [default] [JOB 77] Flush lasted 520136 microseconds, and 445950 cpu microseconds. 2024/11/19-15:17:32.957663 139850957195008 (Original Log Time 2024/11/19-15:17:32.933456) [flush_job.cc:1029] [default] [JOB 77] Level-0 flush table #4579: 541066010 bytes OK (needs compaction) 2024/11/19-15:17:32.957674 139850957195008 (Original Log Time 2024/11/19-15:17:32.934575) [memtable_list.cc:560] [default] Level-0 commit table #4579 started 2024/11/19-15:17:32.957678 139850957195008 (Original Log Time 2024/11/19-15:17:32.957589) [memtable_list.cc:759] [default] Level-0 commit table #4579: memtable #1 done 2024/11/19-15:17:32.957680 139850957195008 (Original Log Time 2024/11/19-15:17:32.957614) EVENT_LOG_v1 {"time_micros": 1732000652957609, "job": 77, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [2, 89, 10, 44, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:17:32.957683 139850957195008 (Original Log Time 2024/11/19-15:17:32.957635) [db_impl_compaction_flush.cc:310] [default] Level summary: files[2 89 10 44 0 0 0] max score 3.92 (73 files need compaction) 2024/11/19-15:17:32.990278 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-02 2024/11/19-15:17:32.990373 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00069/att-02/compact.done) = No such file or directory: 7.193 sec on http probe failed: 3, retry 2024/11/19-15:17:33.352313 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-01 2024/11/19-15:17:33.352401 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00071/att-01/compact.done) = No such file or directory: 6.899 sec on http probe failed: 3, retry 2024/11/19-15:17:33.656555 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-02 2024/11/19-15:17:33.656629 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00072/att-02/compact.done) = No such file or directory: 2.585 sec on http probe failed: 2, retry 2024/11/19-15:17:33.973445 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-00 2024/11/19-15:17:33.973569 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00076/att-00/compact.done) = No such file or directory: 11.220 sec on http probe failed: 5, giveup 2024/11/19-15:17:33.973597 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00076/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.821 sec, real 11.501 (submit 271.347 ms wait 11.220 s), input 127.821 MiB, send shutdown 2024/11/19-15:17:33.982498 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-00 2024/11/19-15:17:33.982801 139851127064320 [dcompact_etcd.cc:1773] job-00076/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:34.040776 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-00 2024/11/19-15:17:34.040844 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00075/att-00/compact.done) = No such file or directory: 13.033 sec on http probe failed: 5, giveup 2024/11/19-15:17:34.040863 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00075/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 248.724 sec, real 13.345 (submit 308.480 ms wait 13.033 s), input 248.724 MiB, send shutdown 2024/11/19-15:17:34.049311 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-00 2024/11/19-15:17:34.049501 139851148035840 [dcompact_etcd.cc:1773] job-00075/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:34.265291 139851127064320 [dcompact_etcd.cc:1163] job-00076/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.282 sec, input = 127.821 MiB 2024/11/19-15:17:34.348401 139851148035840 [dcompact_etcd.cc:1163] job-00075/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.299 sec, input = 248.724 MiB 2024/11/19-15:17:34.538909 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-01 2024/11/19-15:17:34.539004 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00074/att-01/compact.done) = No such file or directory: 2.849 sec on http probe failed: 1, retry 2024/11/19-15:17:34.808947 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-02 2024/11/19-15:17:34.809047 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00072/att-02/compact.done) = No such file or directory: 3.734 sec on http probe failed: 3, retry 2024/11/19-15:17:34.898927 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-01 2024/11/19-15:17:34.899010 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00073/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:17:34.999947 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-02 2024/11/19-15:17:35.000009 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00069/att-02/compact.done) = No such file or directory: 9.202 sec on http probe failed: 4, retry 2024/11/19-15:17:35.357064 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-01 2024/11/19-15:17:35.357160 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00071/att-01/compact.done) = No such file or directory: 8.907 sec on http probe failed: 4, retry 2024/11/19-15:17:35.958660 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-02 2024/11/19-15:17:35.958761 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00072/att-02/compact.done) = No such file or directory: 4.887 sec on http probe failed: 4, retry 2024/11/19-15:17:36.548119 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-01 2024/11/19-15:17:36.548217 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00074/att-01/compact.done) = No such file or directory: 4.858 sec on http probe failed: 2, retry 2024/11/19-15:17:36.907692 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-01 2024/11/19-15:17:36.907794 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00073/att-01/compact.done) = No such file or directory: 5.204 sec on http probe failed: 2, retry 2024/11/19-15:17:37.008045 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-02 2024/11/19-15:17:37.008141 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00069/att-02/compact.done) = No such file or directory: 11.212 sec on http probe failed: 5, giveup 2024/11/19-15:17:37.008160 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00069/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.237 sec, real 11.517 (submit 296.582 ms wait 11.212 s), input 127.237 MiB, send shutdown 2024/11/19-15:17:37.013605 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00069/att-02 2024/11/19-15:17:37.013823 139851179493120 [dcompact_etcd.cc:1773] job-00069/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:37.015696 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00069) = success 2024/11/19-15:17:37.015703 139851179493120 [dcompact_etcd.cc:1773] job-00069/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:17:37.015722 139851179493120 (Original Log Time 2024/11/19-15:17:02.475104) EVENT_LOG_v1 {"time_micros": 1732000622475093, "job": 69, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4478, 4480, 4482, 4484], "score": 5.23364, "input_data_size": 134027720, "oldest_snapshot_seqno": 114167388} 2024/11/19-15:17:37.015730 139851179493120 [compaction_job.cc:2437] [default] [JOB 69] Compacting 4@1 files to L2, score 5.23, subcompactions 1 : 1 2024/11/19-15:17:37.015738 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 67 Base level 1, inputs: [4478(31MB) 4480(31MB) 4482(31MB) 4484(31MB)] 2024/11/19-15:17:37.015961 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004580.sst) /storage/mytopling/..//tmp/Topling-82038-Dy5rX4 2024/11/19-15:17:37.090389 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004580.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:17:37.095694 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004580.sst): index pass time = 0.01's, 228.308'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 100336 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:17:37.111832 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-02 2024/11/19-15:17:37.111933 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00072/att-02/compact.done) = No such file or directory: 6.037 sec on http probe failed: 5, giveup 2024/11/19-15:17:37.111950 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00072/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 57.222 sec, real 6.345 (submit 300.040 ms wait 6.037 s), input 57.222 MiB, send shutdown 2024/11/19-15:17:37.120234 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00072/att-02 2024/11/19-15:17:37.120481 139850978166528 [dcompact_etcd.cc:1773] job-00072/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:37.122257 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00072) = success 2024/11/19-15:17:37.122272 139850978166528 [dcompact_etcd.cc:1773] job-00072/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:17:37.122294 139850978166528 (Original Log Time 2024/11/19-15:17:18.074840) EVENT_LOG_v1 {"time_micros": 1732000638074832, "job": 72, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4491, 4472, 4481, 4483], "score": 4.75752, "input_data_size": 62834780, "oldest_snapshot_seqno": 115513089} 2024/11/19-15:17:37.122302 139850978166528 [compaction_job.cc:2437] [default] [JOB 72] Compacting 4@1 files to L2, score 4.76, subcompactions 1 : 1 2024/11/19-15:17:37.122310 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 70 Base level 1, inputs: [4491(4758KB) 4472(31MB) 4481(3675KB) 4483(20MB)] 2024/11/19-15:17:37.122521 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004581.sst) /storage/mytopling/..//tmp/Topling-82038-Bwnt6L 2024/11/19-15:17:37.158857 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004580.sst): first pass time = 0.14's, 469.672'MB/sec 2024/11/19-15:17:37.158908 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004580.sst): old prealloc_size = 73819750, real_size = 65222960 2024/11/19-15:17:37.158986 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004580.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 100336 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0650 GB avg-val = 644.85 avg-zval = 648.10 2024/11/19-15:17:37.159029 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004580.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.069 sec, Key+Value = 0.067 GB 2024/11/19-15:17:37.225885 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004580.sst): second pass time = 0.00's,192563205.357'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 973.698'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2513.698'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.572'MB/sec write SST data time = 0.02's, 2832.503'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 192563205.357'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 100336 avg-key = 24.00 avg-zkey = 1.70 avg-val = 644.85 avg-zval = 648.10 usrkeys = 100336 avg-key = 16.00 avg-zkey = 1.70 avg-val = 644.85 avg-zval = 648.10 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7942 UnZipSize{ index = 0.0016 GB value = 0.0647 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0002 GB value = 0.0650 GB dict = 0.00 MB all = 0.0652 GB } UnZip/Zip{ index = 9.4372 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9715 } ---------------------------- total value len = 2.584881 GB avg = 0.437 KB (by entry num) total key len = 0.146422 GB avg = 0.025 KB total ukey len = 0.099087 GB avg = 0.017 KB total ukey num = 0.005916917 Billion total entry num = 0.005916917 Billion write speed all = 16.808513415 MB/sec (with seq num) write speed all = 16.517210500 MB/sec (without seq num) 2024/11/19-15:17:37.229701 139851179493120 [compaction_job.cc:1948] [default] [JOB 69] Generated table #4580: 100336 keys, 65199757 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:37.229753 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000657229731, "cf_name": "default", "job": 69, "event": "table_file_creation", "file_number": 4580, "file_size": 65199757, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 65027552, "index_size": 170112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2408064, "raw_average_key_size": 24, "raw_value_size": 64701237, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 100336, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000657, "oldest_key_time": 0, "file_creation_time": 1732000657, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4580, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:17:37.244222 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004582.sst) /storage/mytopling/..//tmp/Topling-82038-6w5ld3 2024/11/19-15:17:37.282612 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004581.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.036 GB 2024/11/19-15:17:37.282701 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004581.sst): index pass time = 0.00's,63725.978'MB/sec index type = UintIndex_AllOne usrkeys = 480000 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0038 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:17:37.302888 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004582.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:17:37.308153 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004582.sst): index pass time = 0.01's, 226.744'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 99078 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:17:37.332541 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004581.sst): first pass time = 0.21's, 169.144'MB/sec 2024/11/19-15:17:37.332587 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004581.sst): old prealloc_size = 69118258, real_size = 27542328 2024/11/19-15:17:37.332610 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004581.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 480000 prefix = 4 raw-key = 0.0038 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0259 GB zip-val = 0.0274 GB avg-val = 54.00 avg-zval = 57.13 2024/11/19-15:17:37.361977 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-01 2024/11/19-15:17:37.362068 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00071/att-01/compact.done) = No such file or directory: 10.912 sec on http probe failed: 5, giveup 2024/11/19-15:17:37.362093 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00071/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 115.378 sec, real 11.188 (submit 271.477 ms wait 10.912 s), input 115.378 MiB, send shutdown 2024/11/19-15:17:37.367096 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-01 2024/11/19-15:17:37.367309 139851169007360 [dcompact_etcd.cc:1773] job-00071/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:37.370736 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004582.sst): first pass time = 0.13's, 524.179'MB/sec 2024/11/19-15:17:37.370778 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004582.sst): old prealloc_size = 73819750, real_size = 64445328 2024/11/19-15:17:37.370846 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004582.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 99078 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0639 GB zip-val = 0.0643 GB avg-val = 645.25 avg-zval = 648.50 2024/11/19-15:17:37.370886 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004582.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.068 sec, Key+Value = 0.066 GB 2024/11/19-15:17:37.379860 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004581.sst): second pass time = 0.00's,73641315.341'MB/sec, value only(73.0% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 580.260'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.150'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 6.947'MB/sec write SST data time = 0.03's, 915.048'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.026'GB zip my value throughput = 73641315.341'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 480000 avg-key = 20.00 avg-zkey = 0.00 avg-val = 54.00 avg-zval = 57.13 usrkeys = 480000 avg-key = 12.00 avg-zkey = 0.00 avg-val = 54.00 avg-zval = 57.13 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0058 GB value = 0.0259 GB dict = 0.00 MB all = 0.0355 GB } __ZipSize{ index = 0.0000 GB value = 0.0274 GB dict = 0.00 MB all = 0.0274 GB } UnZip/Zip{ index =30000.0000 value = 0.9453 dict = 0.00 all = 1.2953 } Zip/UnZip{ index = 0.0000 value = 1.0579 dict = 0.00 all = 0.7720 } ---------------------------- total value len = 2.610803 GB avg = 0.408 KB (by entry num) total key len = 0.156022 GB avg = 0.024 KB total ukey len = 0.104847 GB avg = 0.016 KB total ukey num = 0.006396917 Billion total entry num = 0.006396917 Billion write speed all = 17.010995559 MB/sec (with seq num) write speed all = 16.696359331 MB/sec (without seq num) 2024/11/19-15:17:37.381724 139850978166528 [compaction_job.cc:1948] [default] [JOB 72] Generated table #4581: 480000 keys, 27423240 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:37.381773 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000657381750, "cf_name": "default", "job": 72, "event": "table_file_creation", "file_number": 4581, "file_size": 27423240, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27421952, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 9600000, "raw_average_key_size": 20, "raw_value_size": 25921743, "raw_average_value_size": 54, "num_data_blocks": 1, "num_entries": 480000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000657, "oldest_key_time": 0, "file_creation_time": 1732000657, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4581, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:17:37.389407 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004583.sst) /storage/mytopling/..//tmp/Topling-82038-33NRpO 2024/11/19-15:17:37.424092 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004583.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0013 GB, waited 0.000 sec, Key+Value = 0.003 GB 2024/11/19-15:17:37.426917 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004583.sst): first pass time = 0.04's, 92.173'MB/sec 2024/11/19-15:17:37.432546 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004583.sst): index pass time = 0.01's, 205.303'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 144000 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0017 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:17:37.432616 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004583.sst): old prealloc_size = 69118258, real_size = 280416 2024/11/19-15:17:37.432695 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004583.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZeroLengthBlobStore usrkeys = 144000 prefix = 4 raw-key = 0.0017 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 0.00 avg-zval = 0.00 2024/11/19-15:17:37.432733 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004583.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.009 sec, Key+Value = 0.003 GB 2024/11/19-15:17:37.437758 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004582.sst): second pass time = 0.00's,217449442.177'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 961.994'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2559.961'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.570'MB/sec write SST data time = 0.02's, 2748.533'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.064'GB zip my value throughput = 217449442.177'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 99078 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.25 avg-zval = 648.50 usrkeys = 99078 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.25 avg-zval = 648.50 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7847 UnZipSize{ index = 0.0016 GB value = 0.0639 GB dict = 0.00 MB all = 0.0663 GB } __ZipSize{ index = 0.0002 GB value = 0.0643 GB dict = 0.00 MB all = 0.0644 GB } UnZip/Zip{ index = 9.4360 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 2.674733 GB avg = 0.412 KB (by entry num) total key len = 0.158400 GB avg = 0.024 KB total ukey len = 0.106432 GB avg = 0.016 KB total ukey num = 0.006495995 Billion total entry num = 0.006495995 Billion write speed all = 17.412472101 MB/sec (with seq num) write speed all = 17.093076359 MB/sec (without seq num) 2024/11/19-15:17:37.441671 139851179493120 [compaction_job.cc:1948] [default] [JOB 69] Generated table #4582: 99078 keys, 64422413 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:37.441722 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000657441698, "cf_name": "default", "job": 69, "event": "table_file_creation", "file_number": 4582, "file_size": 64422413, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64252352, "index_size": 168000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2377872, "raw_average_key_size": 24, "raw_value_size": 63930136, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 99078, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000657, "oldest_key_time": 0, "file_creation_time": 1732000657, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4582, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:17:37.449590 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004583.sst): second pass time = 0.00's, 0.000'MB/sec, value only( 0.0% of KV) wait indexing time = 0.01's, remap KeyValue time = 0.02's, 14.514'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3364.244'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 151.633'MB/sec write SST data time = 0.02's, 14.783'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 0.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 144000 avg-key = 24.00 avg-zkey = 1.70 avg-val = 0.00 avg-zval = 0.00 usrkeys = 144000 avg-key = 16.00 avg-zkey = 1.70 avg-val = 0.00 avg-zval = 0.00 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 5 UnZipSize{ index = 0.0023 GB value = 0.0000 GB dict = 0.00 MB all = 0.0035 GB } __ZipSize{ index = 0.0002 GB value = 0.0000 GB dict = 0.00 MB all = 0.0002 GB } UnZip/Zip{ index = 9.4364 value = 0.0000 dict = 0.00 all = 14.0318 } Zip/UnZip{ index = 0.1060 value = inf dict = 0.00 all = 0.0713 } ---------------------------- total value len = 2.674733 GB avg = 0.403 KB (by entry num) total key len = 0.161856 GB avg = 0.024 KB total ukey len = 0.108736 GB avg = 0.016 KB total ukey num = 0.006639995 Billion total entry num = 0.006639995 Billion write speed all = 17.432444650 MB/sec (with seq num) write speed all = 17.105992448 MB/sec (without seq num) 2024/11/19-15:17:37.449901 139850978166528 [compaction_job.cc:1948] [default] [JOB 72] Generated table #4583: 144000 keys, 246297 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:37.449952 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000657449930, "cf_name": "default", "job": 72, "event": "table_file_creation", "file_number": 4583, "file_size": 246297, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 80, "index_size": 244160, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3456000, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 144000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZeroLengthBlobStore", "creation_time": 1732000657, "oldest_key_time": 0, "file_creation_time": 1732000657, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4583, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:17:37.450517 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004584.sst) /storage/mytopling/..//tmp/Topling-82038-wMUpSY 2024/11/19-15:17:37.456340 139851179493120 [compaction_job.cc:703] job-00069: subcompact[0], size: 0.129622 G, files: /storage/mytopling/.rocksdb [004580,004582] 2024/11/19-15:17:37.457263 139851179493120 (Original Log Time 2024/11/19-15:17:37.015771) EVENT_LOG_v1 {"time_micros": 1732000657015751, "job": 69, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4478, 4480, 4482, 4484], "score": 5.23364, "input_data_size": 134027720, "oldest_snapshot_seqno": 114167388} 2024/11/19-15:17:37.457273 139851179493120 (Original Log Time 2024/11/19-15:17:37.456803) [compaction_job.cc:2022] [default] [JOB 69] Compacted 4@1 files to L2 => 129622170 bytes 2024/11/19-15:17:37.457276 139851179493120 (Original Log Time 2024/11/19-15:17:37.457139) [compaction_job.cc:1224] [default] compacted to: files[2 85 12 44 0 0 0] max score 3.92 (75 files need compaction), MB/sec: 304.2 rd, 294.2 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.8, 0.0 +0.0 blob) out(123.6 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 199414, records dropped: 0 output_compression: Snappy 2024/11/19-15:17:37.457295 139851179493120 (Original Log Time 2024/11/19-15:17:37.457164) EVENT_LOG_v1 {"time_micros": 1732000657457150, "cf": "default", "job": 69, "event": "compaction_finished", "compaction_time_micros": 440580, "compaction_time_cpu_micros": 438652, "output_level": 2, "num_output_files": 2, "total_output_size": 129622170, "num_input_records": 199414, "num_output_records": 199414, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 85, 12, 44, 0, 0, 0]} 2024/11/19-15:17:37.463875 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004437.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:37.463899 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000657463896, "job": 69, "event": "table_file_deletion", "file_number": 4437} 2024/11/19-15:17:37.467129 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-01 2024/11/19-15:17:37.467200 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00076/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:17:37.470868 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004438.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:37.470891 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000657470889, "job": 69, "event": "table_file_deletion", "file_number": 4438} 2024/11/19-15:17:37.477419 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004439.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:37.477441 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000657477438, "job": 69, "event": "table_file_deletion", "file_number": 4439} 2024/11/19-15:17:37.483290 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004440.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:37.483312 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000657483310, "job": 69, "event": "table_file_deletion", "file_number": 4440} 2024/11/19-15:17:37.483475 139851179493120 [compaction_job.cc:2437] [default] [JOB 78] Compacting 4@1 files to L2, score 3.92, subcompactions 1 : 1 2024/11/19-15:17:37.483484 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 76 Base level 1, inputs: [4497(31MB) 4500(31MB) 4505(31MB) 4509(31MB)] 2024/11/19-15:17:37.573987 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004584.sst: sumWaitingMem = 0.004 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0043 GB, waited 0.000 sec, Key+Value = 0.021 GB 2024/11/19-15:17:37.614750 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004584.sst): index pass time = 0.04's, 141.444'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 480000 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0058 GB zip-key = 0.0008 GB avg-key = 12.00 avg-zkey = 1.68 2024/11/19-15:17:37.618480 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004584.sst): first pass time = 0.17's, 125.180'MB/sec 2024/11/19-15:17:37.618524 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004584.sst): old prealloc_size = 69118258, real_size = 10903568 2024/11/19-15:17:37.618727 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004584.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 480000 prefix = 4 raw-key = 0.0058 GB zip-key = 0.0008 GB avg-key = 12.00 avg-zkey = 1.68 raw-val = 0.0095 GB zip-val = 0.0100 GB avg-val = 19.80 avg-zval = 20.78 2024/11/19-15:17:37.618767 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004584.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.045 sec, Key+Value = 0.021 GB 2024/11/19-15:17:37.663315 139851169007360 [dcompact_etcd.cc:1163] job-00071/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.296 sec, input = 115.378 MiB 2024/11/19-15:17:37.675334 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004584.sst): second pass time = 0.00's,49500000.000'MB/sec, value only(45.2% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 189.949'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 4022.340'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 7.110'MB/sec write SST data time = 0.04's, 271.620'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.010'GB zip my value throughput = 49500000.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 480000 avg-key = 24.00 avg-zkey = 1.68 avg-val = 19.80 avg-zval = 20.78 usrkeys = 480000 avg-key = 16.00 avg-zkey = 1.68 avg-val = 19.80 avg-zval = 20.78 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 1167 UnZipSize{ index = 0.0077 GB value = 0.0095 GB dict = 0.00 MB all = 0.0210 GB } __ZipSize{ index = 0.0008 GB value = 0.0100 GB dict = 0.00 MB all = 0.0108 GB } UnZip/Zip{ index = 9.5042 value = 0.9528 dict = 0.00 all = 1.9487 } Zip/UnZip{ index = 0.1052 value = 1.0496 dict = 0.00 all = 0.5132 } ---------------------------- total value len = 2.684237 GB avg = 0.377 KB (by entry num) total key len = 0.173376 GB avg = 0.024 KB total ukey len = 0.116416 GB avg = 0.016 KB total ukey num = 0.007119995 Billion total entry num = 0.007119995 Billion write speed all = 17.537319551 MB/sec (with seq num) write speed all = 17.187753329 MB/sec (without seq num) 2024/11/19-15:17:37.676314 139850978166528 [compaction_job.cc:1948] [default] [JOB 72] Generated table #4584: 480000 keys, 10788496 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:37.676366 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000657676342, "cf_name": "default", "job": 72, "event": "table_file_creation", "file_number": 4584, "file_size": 10788496, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 9975296, "index_size": 808064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 11520000, "raw_average_key_size": 24, "raw_value_size": 9504000, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 480000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000657, "oldest_key_time": 0, "file_creation_time": 1732000657, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4584, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:17:37.680210 139850978166528 [compaction_job.cc:703] job-00072: subcompact[0], size: 0.038458 G, files: /storage/mytopling/.rocksdb [004581,004583,004584] 2024/11/19-15:17:37.681403 139850978166528 (Original Log Time 2024/11/19-15:17:37.122343) EVENT_LOG_v1 {"time_micros": 1732000657122323, "job": 72, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4491, 4472, 4481, 4483], "score": 4.75752, "input_data_size": 62834780, "oldest_snapshot_seqno": 115513089} 2024/11/19-15:17:37.681413 139850978166528 (Original Log Time 2024/11/19-15:17:37.681024) [compaction_job.cc:2022] [default] [JOB 72] Compacted 4@1 files to L2 => 38458033 bytes 2024/11/19-15:17:37.681416 139850978166528 (Original Log Time 2024/11/19-15:17:37.681291) [compaction_job.cc:1224] [default] compacted to: files[2 81 15 44 0 0 0] max score 3.67 (74 files need compaction), MB/sec: 112.6 rd, 68.9 wr, level 2, files in(4, 0) out(3 +0 blob) MB in(59.9, 0.0 +0.0 blob) out(36.7 +0.0 blob), read-write-amplify(1.6) write-amplify(0.6) OK, records in: 1104000, records dropped: 0 output_compression: Snappy 2024/11/19-15:17:37.681430 139850978166528 (Original Log Time 2024/11/19-15:17:37.681317) EVENT_LOG_v1 {"time_micros": 1732000657681304, "cf": "default", "job": 72, "event": "compaction_finished", "compaction_time_micros": 557879, "compaction_time_cpu_micros": 542436, "output_level": 2, "num_output_files": 3, "total_output_size": 38458033, "num_input_records": 1104000, "num_output_records": 1104000, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 81, 15, 44, 0, 0, 0]} 2024/11/19-15:17:37.681502 139850978166528 [compaction_job.cc:2437] [default] [JOB 79] Compacting 4@1 files to L2, score 3.67, subcompactions 1 : 1 2024/11/19-15:17:37.681510 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 77 Base level 1, inputs: [4510(31MB) 4511(31MB) 4512(31MB) 4513(31MB)] 2024/11/19-15:17:37.774544 139851179493120 [dcompact_etcd.cc:1163] job-00078/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.291 sec, input = 127.819 MiB 2024/11/19-15:17:37.953910 139850978166528 [dcompact_etcd.cc:1163] job-00079/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.272 sec, input = 127.820 MiB 2024/11/19-15:17:38.554943 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-01 2024/11/19-15:17:38.555030 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00074/att-01/compact.done) = No such file or directory: 6.867 sec on http probe failed: 3, retry 2024/11/19-15:17:38.912748 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-01 2024/11/19-15:17:38.912821 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00073/att-01/compact.done) = No such file or directory: 7.213 sec on http probe failed: 3, retry 2024/11/19-15:17:39.353298 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-01 2024/11/19-15:17:39.353398 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00075/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:17:39.475658 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-01 2024/11/19-15:17:39.475741 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00076/att-01/compact.done) = No such file or directory: 5.202 sec on http probe failed: 2, retry 2024/11/19-15:17:40.554504 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-02 2024/11/19-15:17:40.554588 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00071/att-02/compact.done) = No such file or directory: 2.885 sec on http probe failed: 1, retry 2024/11/19-15:17:40.563582 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-01 2024/11/19-15:17:40.563633 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00074/att-01/compact.done) = No such file or directory: 8.874 sec on http probe failed: 4, retry 2024/11/19-15:17:40.921709 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-01 2024/11/19-15:17:40.921801 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00073/att-01/compact.done) = No such file or directory: 9.218 sec on http probe failed: 4, retry 2024/11/19-15:17:40.975027 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-00 2024/11/19-15:17:40.975099 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00078/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:17:41.157815 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-00 2024/11/19-15:17:41.157912 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00079/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:17:41.361741 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-01 2024/11/19-15:17:41.361811 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00075/att-01/compact.done) = No such file or directory: 7.005 sec on http probe failed: 2, retry 2024/11/19-15:17:41.485861 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-01 2024/11/19-15:17:41.485983 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00076/att-01/compact.done) = No such file or directory: 7.211 sec on http probe failed: 3, retry 2024/11/19-15:17:42.561654 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-02 2024/11/19-15:17:42.561794 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00071/att-02/compact.done) = No such file or directory: 4.891 sec on http probe failed: 2, retry 2024/11/19-15:17:42.572821 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-01 2024/11/19-15:17:42.572903 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00074/att-01/compact.done) = No such file or directory: 10.882 sec on http probe failed: 5, giveup 2024/11/19-15:17:42.572942 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00074/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 113.940 sec, real 11.145 (submit 253.093 ms wait 10.882 s), input 113.940 MiB, send shutdown 2024/11/19-15:17:42.577420 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-01 2024/11/19-15:17:42.578469 139851137550080 [dcompact_etcd.cc:1773] job-00074/att-01: CleanFiles 0.001 sec : 0.000 2024/11/19-15:17:42.892226 139851137550080 [dcompact_etcd.cc:1163] job-00074/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.314 sec, input = 113.940 MiB 2024/11/19-15:17:42.926623 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-01 2024/11/19-15:17:42.926727 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00073/att-01/compact.done) = No such file or directory: 11.227 sec on http probe failed: 5, giveup 2024/11/19-15:17:42.926753 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00073/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.818 sec, real 11.501 (submit 269.140 ms wait 11.227 s), input 127.818 MiB, send shutdown 2024/11/19-15:17:42.936849 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-01 2024/11/19-15:17:42.937150 139851158521600 [dcompact_etcd.cc:1773] job-00073/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:42.983101 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-00 2024/11/19-15:17:42.983161 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00078/att-00/compact.done) = No such file or directory: 5.201 sec on http probe failed: 2, retry 2024/11/19-15:17:43.163557 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-00 2024/11/19-15:17:43.163648 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00079/att-00/compact.done) = No such file or directory: 5.204 sec on http probe failed: 2, retry 2024/11/19-15:17:43.216962 139851158521600 [dcompact_etcd.cc:1163] job-00073/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 127.818 MiB 2024/11/19-15:17:43.370715 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-01 2024/11/19-15:17:43.370808 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00075/att-01/compact.done) = No such file or directory: 9.013 sec on http probe failed: 3, retry 2024/11/19-15:17:43.494464 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-01 2024/11/19-15:17:43.494547 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00076/att-01/compact.done) = No such file or directory: 9.221 sec on http probe failed: 4, retry 2024/11/19-15:17:44.567088 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-02 2024/11/19-15:17:44.567174 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00071/att-02/compact.done) = No such file or directory: 6.899 sec on http probe failed: 3, retry 2024/11/19-15:17:44.989990 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-00 2024/11/19-15:17:44.990071 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00078/att-00/compact.done) = No such file or directory: 7.209 sec on http probe failed: 3, retry 2024/11/19-15:17:45.172080 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-00 2024/11/19-15:17:45.172226 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00079/att-00/compact.done) = No such file or directory: 7.210 sec on http probe failed: 3, retry 2024/11/19-15:17:45.379229 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-01 2024/11/19-15:17:45.379313 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00075/att-01/compact.done) = No such file or directory: 11.022 sec on http probe failed: 4, retry 2024/11/19-15:17:45.501852 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-01 2024/11/19-15:17:45.501955 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00076/att-01/compact.done) = No such file or directory: 11.229 sec on http probe failed: 5, giveup 2024/11/19-15:17:45.501974 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00076/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.821 sec, real 11.519 (submit 282.327 ms wait 11.229 s), input 127.821 MiB, send shutdown 2024/11/19-15:17:45.510344 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-01 2024/11/19-15:17:45.659761 139851127064320 [dcompact_etcd.cc:1773] job-00076/att-01: CleanFiles 0.149 sec : 0.000 2024/11/19-15:17:45.746329 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-02 2024/11/19-15:17:45.746424 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00074/att-02/compact.done) = No such file or directory: 2.849 sec on http probe failed: 1, retry 2024/11/19-15:17:45.952662 139851127064320 [dcompact_etcd.cc:1163] job-00076/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.293 sec, input = 127.821 MiB 2024/11/19-15:17:46.418221 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-02 2024/11/19-15:17:46.418331 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00073/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:17:46.575295 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-02 2024/11/19-15:17:46.575372 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00071/att-02/compact.done) = No such file or directory: 8.904 sec on http probe failed: 4, retry 2024/11/19-15:17:46.999110 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-00 2024/11/19-15:17:46.999211 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00078/att-00/compact.done) = No such file or directory: 9.216 sec on http probe failed: 4, retry 2024/11/19-15:17:47.179000 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-00 2024/11/19-15:17:47.179086 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00079/att-00/compact.done) = No such file or directory: 9.218 sec on http probe failed: 4, retry 2024/11/19-15:17:47.385089 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-01 2024/11/19-15:17:47.385186 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00075/att-01/compact.done) = No such file or directory: 13.031 sec on http probe failed: 5, giveup 2024/11/19-15:17:47.385205 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00075/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 248.724 sec, real 13.336 (submit 298.825 ms wait 13.031 s), input 248.724 MiB, send shutdown 2024/11/19-15:17:47.393720 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-01 2024/11/19-15:17:47.393963 139851148035840 [dcompact_etcd.cc:1773] job-00075/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:47.683209 139851148035840 [dcompact_etcd.cc:1163] job-00075/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 248.724 MiB 2024/11/19-15:17:47.751443 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-02 2024/11/19-15:17:47.751511 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00074/att-02/compact.done) = No such file or directory: 4.854 sec on http probe failed: 2, retry 2024/11/19-15:17:48.195891 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4585. Immutable memtables: 0. 2024/11/19-15:17:48.196609 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 80] Syncing log #4578 2024/11/19-15:17:48.196701 139850967680768 (Original Log Time 2024/11/19-15:17:48.196383) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:17:48.196709 139850967680768 [flush_job.cc:879] [default] [JOB 80] Flushing memtable with next log file: 4585 2024/11/19-15:17:48.196729 139850967680768 EVENT_LOG_v1 {"time_micros": 1732000668196724, "job": 80, "event": "flush_started", "num_memtables": 1, "num_entries": 1167193, "num_deletes": 0, "total_data_size": 478126606, "memory_usage": 545521032, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:17:48.196733 139850967680768 [flush_job.cc:910] [default] [JOB 80] Level-0 flush table #4586: started 2024/11/19-15:17:48.197333 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000008.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004586.sst in ConvertToSST, mem_size = 520.249 M, time = 590.992 us 2024/11/19-15:17:48.302339 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004586.sst): fsize = 520.251 M, time(ms): open: 0.033, seek: 0.013, finish: 0.037, meta: 0.012, Flush: 0.013, sync: 104.870, close: 0.011, all: 104.989 2024/11/19-15:17:48.303442 139850967680768 [flush_job.cc:1083] [default] [JOB 80] Flush lasted 106740 microseconds, and 88448 cpu microseconds. 2024/11/19-15:17:48.325001 139850967680768 (Original Log Time 2024/11/19-15:17:48.302383) [flush_job.cc:1029] [default] [JOB 80] Level-0 flush table #4586: 545522458 bytes OK (needs compaction) 2024/11/19-15:17:48.325012 139850967680768 (Original Log Time 2024/11/19-15:17:48.303458) [memtable_list.cc:560] [default] Level-0 commit table #4586 started 2024/11/19-15:17:48.325015 139850967680768 (Original Log Time 2024/11/19-15:17:48.324928) [memtable_list.cc:759] [default] Level-0 commit table #4586: memtable #1 done 2024/11/19-15:17:48.325018 139850967680768 (Original Log Time 2024/11/19-15:17:48.324953) EVENT_LOG_v1 {"time_micros": 1732000668324948, "job": 80, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [3, 81, 15, 44, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:17:48.325022 139850967680768 (Original Log Time 2024/11/19-15:17:48.324972) [db_impl_compaction_flush.cc:310] [default] Level summary: files[3 81 15 44 0 0 0] max score 3.42 (71 files need compaction) 2024/11/19-15:17:48.423765 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-02 2024/11/19-15:17:48.423851 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00073/att-02/compact.done) = No such file or directory: 5.201 sec on http probe failed: 2, retry 2024/11/19-15:17:48.584747 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-02 2024/11/19-15:17:48.584821 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00071/att-02/compact.done) = No such file or directory: 10.912 sec on http probe failed: 5, giveup 2024/11/19-15:17:48.584838 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00071/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 115.378 sec, real 11.217 (submit 295.934 ms wait 10.912 s), input 115.378 MiB, send shutdown 2024/11/19-15:17:48.595192 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00071/att-02 2024/11/19-15:17:48.595417 139851169007360 [dcompact_etcd.cc:1773] job-00071/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:48.597369 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00071) = success 2024/11/19-15:17:48.597377 139851169007360 [dcompact_etcd.cc:1773] job-00071/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:17:48.597396 139851169007360 (Original Log Time 2024/11/19-15:17:14.939296) EVENT_LOG_v1 {"time_micros": 1732000634939287, "job": 71, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4486, 4487, 4488, 4490], "score": 4.98399, "input_data_size": 121586248, "oldest_snapshot_seqno": -1} 2024/11/19-15:17:48.597403 139851169007360 [compaction_job.cc:2437] [default] [JOB 71] Compacting 4@1 files to L2, score 4.98, subcompactions 1 : 1 2024/11/19-15:17:48.597411 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 69 Base level 1, inputs: [4486(31MB) 4487(31MB) 4488(31MB) 4490(20MB)] 2024/11/19-15:17:48.597625 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004587.sst) /storage/mytopling/..//tmp/Topling-82038-oAPkSS 2024/11/19-15:17:48.656666 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004587.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:17:48.662000 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004587.sst): index pass time = 0.01's, 227.163'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 100339 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:17:48.725015 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004587.sst): first pass time = 0.13's, 526.858'MB/sec 2024/11/19-15:17:48.725069 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004587.sst): old prealloc_size = 73819750, real_size = 65223040 2024/11/19-15:17:48.725140 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004587.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 100339 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0650 GB avg-val = 644.83 avg-zval = 648.08 2024/11/19-15:17:48.725190 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004587.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.069 sec, Key+Value = 0.067 GB 2024/11/19-15:17:48.793391 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004587.sst): second pass time = 0.00's,184860745.714'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 954.548'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2537.025'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.570'MB/sec write SST data time = 0.02's, 2690.333'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 184860745.714'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 100339 avg-key = 24.00 avg-zkey = 1.70 avg-val = 644.83 avg-zval = 648.08 usrkeys = 100339 avg-key = 16.00 avg-zkey = 1.70 avg-val = 644.83 avg-zval = 648.08 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7942 UnZipSize{ index = 0.0016 GB value = 0.0647 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0002 GB value = 0.0650 GB dict = 0.00 MB all = 0.0652 GB } UnZip/Zip{ index = 9.4339 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9715 } ---------------------------- total value len = 2.748939 GB avg = 0.381 KB (by entry num) total key len = 0.175784 GB avg = 0.024 KB total ukey len = 0.118021 GB avg = 0.016 KB total ukey num = 0.007220334 Billion total entry num = 0.007220334 Billion write speed all = 16.802691359 MB/sec (with seq num) write speed all = 16.470841637 MB/sec (without seq num) 2024/11/19-15:17:48.797072 139851169007360 [compaction_job.cc:1948] [default] [JOB 71] Generated table #4587: 100339 keys, 65199821 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:48.797126 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000668797102, "cf_name": "default", "job": 71, "event": "table_file_creation", "file_number": 4587, "file_size": 65199821, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 65027568, "index_size": 170176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2408136, "raw_average_key_size": 24, "raw_value_size": 64701261, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 100339, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000668, "oldest_key_time": 0, "file_creation_time": 1732000668, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4587, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:17:48.811514 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004588.sst) /storage/mytopling/..//tmp/Topling-82038-G14Ky3 2024/11/19-15:17:48.858754 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004588.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.054 GB 2024/11/19-15:17:48.865431 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004588.sst): index pass time = 0.01's, 145.631'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 80512 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0001 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:17:48.913474 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004588.sst): first pass time = 0.10's, 528.479'MB/sec 2024/11/19-15:17:48.913528 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004588.sst): old prealloc_size = 73819750, real_size = 52360400 2024/11/19-15:17:48.913590 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004588.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 80512 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0001 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0519 GB zip-val = 0.0522 GB avg-val = 645.14 avg-zval = 648.39 2024/11/19-15:17:48.913641 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004588.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.055 sec, Key+Value = 0.054 GB 2024/11/19-15:17:48.967259 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004588.sst): second pass time = 0.00's,154587282.738'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 974.458'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2303.522'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.573'MB/sec write SST data time = 0.02's, 2827.217'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.052'GB zip my value throughput = 154587282.738'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 80512 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.14 avg-zval = 648.39 usrkeys = 80512 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.14 avg-zval = 648.39 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6376 UnZipSize{ index = 0.0013 GB value = 0.0519 GB dict = 0.00 MB all = 0.0539 GB } __ZipSize{ index = 0.0001 GB value = 0.0522 GB dict = 0.00 MB all = 0.0523 GB } UnZip/Zip{ index = 9.4100 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1063 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 2.800880 GB avg = 0.384 KB (by entry num) total key len = 0.177716 GB avg = 0.024 KB total ukey len = 0.119309 GB avg = 0.016 KB total ukey num = 0.007300846 Billion total entry num = 0.007300846 Billion write speed all = 17.095122078 MB/sec (with seq num) write speed all = 16.759906833 MB/sec (without seq num) 2024/11/19-15:17:48.970612 139851169007360 [compaction_job.cc:1948] [default] [JOB 71] Generated table #4588: 80512 keys, 52342028 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:48.970666 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000668970642, "cf_name": "default", "job": 71, "event": "table_file_creation", "file_number": 4588, "file_size": 52342028, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 52203200, "index_size": 136896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1932288, "raw_average_key_size": 24, "raw_value_size": 51941327, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 80512, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000668, "oldest_key_time": 0, "file_creation_time": 1732000668, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4588, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:17:48.982044 139851169007360 [compaction_job.cc:703] job-00071: subcompact[0], size: 0.117542 G, files: /storage/mytopling/.rocksdb [004587,004588] 2024/11/19-15:17:48.982885 139851169007360 (Original Log Time 2024/11/19-15:17:48.597441) EVENT_LOG_v1 {"time_micros": 1732000668597423, "job": 71, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4486, 4487, 4488, 4490], "score": 4.98399, "input_data_size": 121586248, "oldest_snapshot_seqno": -1} 2024/11/19-15:17:48.982894 139851169007360 (Original Log Time 2024/11/19-15:17:48.982504) [compaction_job.cc:2022] [default] [JOB 71] Compacted 4@1 files to L2 => 117541849 bytes 2024/11/19-15:17:48.982897 139851169007360 (Original Log Time 2024/11/19-15:17:48.982750) [compaction_job.cc:1224] [default] compacted to: files[3 77 17 44 0 0 0] max score 3.42 (73 files need compaction), MB/sec: 316.1 rd, 305.6 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(116.0, 0.0 +0.0 blob) out(112.1 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 180851, records dropped: 0 output_compression: Snappy 2024/11/19-15:17:48.982905 139851169007360 (Original Log Time 2024/11/19-15:17:48.982774) EVENT_LOG_v1 {"time_micros": 1732000668982761, "cf": "default", "job": 71, "event": "compaction_finished", "compaction_time_micros": 384616, "compaction_time_cpu_micros": 381612, "output_level": 2, "num_output_files": 2, "total_output_size": 117541849, "num_input_records": 180851, "num_output_records": 180851, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 77, 17, 44, 0, 0, 0]} 2024/11/19-15:17:48.989133 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004462.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:48.989159 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000668989156, "job": 71, "event": "table_file_deletion", "file_number": 4462} 2024/11/19-15:17:48.995735 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004504.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:48.995759 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000668995756, "job": 71, "event": "table_file_deletion", "file_number": 4504} 2024/11/19-15:17:48.995893 139851169007360 [compaction_job.cc:2437] [default] [JOB 81] Compacting 4@1 files to L2, score 3.42, subcompactions 1 : 1 2024/11/19-15:17:48.995902 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 79 Base level 1, inputs: [4514(31MB) 4515(5657KB) 4516(9050KB) 4517(31MB)] 2024/11/19-15:17:49.008603 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-00 2024/11/19-15:17:49.008688 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00078/att-00/compact.done) = No such file or directory: 11.225 sec on http probe failed: 5, giveup 2024/11/19-15:17:49.008704 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00078/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.819 sec, real 11.525 (submit 290.598 ms wait 11.225 s), input 127.819 MiB, send shutdown 2024/11/19-15:17:49.013545 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-00 2024/11/19-15:17:49.013744 139851179493120 [dcompact_etcd.cc:1773] job-00078/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:49.153858 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-02 2024/11/19-15:17:49.153960 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00076/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:17:49.188214 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-00 2024/11/19-15:17:49.188285 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00079/att-00/compact.done) = No such file or directory: 11.225 sec on http probe failed: 5, giveup 2024/11/19-15:17:49.188303 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00079/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.506 (submit 271.925 ms wait 11.225 s), input 127.820 MiB, send shutdown 2024/11/19-15:17:49.196194 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-00 2024/11/19-15:17:49.196415 139850978166528 [dcompact_etcd.cc:1773] job-00079/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:49.261542 139851179493120 [dcompact_etcd.cc:1163] job-00078/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.248 sec, input = 127.819 MiB 2024/11/19-15:17:49.314735 139851169007360 [dcompact_etcd.cc:1163] job-00081/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.318 sec, input = 77.023 MiB 2024/11/19-15:17:49.494744 139850978166528 [dcompact_etcd.cc:1163] job-00079/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.298 sec, input = 127.820 MiB 2024/11/19-15:17:49.757472 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-02 2024/11/19-15:17:49.757550 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00074/att-02/compact.done) = No such file or directory: 6.859 sec on http probe failed: 3, retry 2024/11/19-15:17:50.432338 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-02 2024/11/19-15:17:50.432428 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00073/att-02/compact.done) = No such file or directory: 7.207 sec on http probe failed: 3, retry 2024/11/19-15:17:51.163982 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-02 2024/11/19-15:17:51.164073 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00076/att-02/compact.done) = No such file or directory: 5.201 sec on http probe failed: 2, retry 2024/11/19-15:17:51.250936 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-00 2024/11/19-15:17:51.251023 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00081/att-00/compact.done) = No such file or directory: 1.926 sec on http probe failed: 1, retry 2024/11/19-15:17:51.765847 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-02 2024/11/19-15:17:51.765945 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00074/att-02/compact.done) = No such file or directory: 8.865 sec on http probe failed: 4, retry 2024/11/19-15:17:52.438176 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-02 2024/11/19-15:17:52.438251 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00073/att-02/compact.done) = No such file or directory: 9.216 sec on http probe failed: 4, retry 2024/11/19-15:17:52.466120 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-01 2024/11/19-15:17:52.466217 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00078/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:17:52.691879 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-02 2024/11/19-15:17:52.691992 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00075/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:17:52.699993 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-01 2024/11/19-15:17:52.700061 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00079/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:17:52.797315 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-00 2024/11/19-15:17:52.797394 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00081/att-00/compact.done) = No such file or directory: 3.477 sec on http probe failed: 2, retry 2024/11/19-15:17:53.173301 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-02 2024/11/19-15:17:53.173379 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00076/att-02/compact.done) = No such file or directory: 7.211 sec on http probe failed: 3, retry 2024/11/19-15:17:53.774108 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-02 2024/11/19-15:17:53.774178 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00074/att-02/compact.done) = No such file or directory: 10.874 sec on http probe failed: 5, giveup 2024/11/19-15:17:53.774196 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00074/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 113.940 sec, real 11.196 (submit 313.631 ms wait 10.874 s), input 113.940 MiB, send shutdown 2024/11/19-15:17:53.779793 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00074/att-02 2024/11/19-15:17:53.780015 139851137550080 [dcompact_etcd.cc:1773] job-00074/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:53.781888 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00074) = success 2024/11/19-15:17:53.781906 139851137550080 [dcompact_etcd.cc:1773] job-00074/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:17:53.781942 139851137550080 (Original Log Time 2024/11/19-15:17:20.262830) EVENT_LOG_v1 {"time_micros": 1732000640262825, "job": 74, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4503, 4506, 4474, 4479], "score": 4.39084, "input_data_size": 118885972, "oldest_snapshot_seqno": -1} 2024/11/19-15:17:53.781953 139851137550080 [compaction_job.cc:2437] [default] [JOB 74] Compacting 4@1 files to L2, score 4.39, subcompactions 1 : 1 2024/11/19-15:17:53.781965 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 72 Base level 1, inputs: [4503(31MB) 4506(18MB) 4474(31MB) 4479(31MB)] 2024/11/19-15:17:53.782259 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004589.sst) /storage/mytopling/..//tmp/Topling-82038-1nIX50 2024/11/19-15:17:53.941461 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004589.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:17:53.975595 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004589.sst): index pass time = 0.03's, 217.467'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463425 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:17:54.042058 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004589.sst): first pass time = 0.26's, 258.327'MB/sec 2024/11/19-15:17:54.042112 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004589.sst): old prealloc_size = 73819750, real_size = 55612112 2024/11/19-15:17:54.042341 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004589.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463425 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.81 avg-zval = 117.86 2024/11/19-15:17:54.042408 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004589.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.101 sec, Key+Value = 0.067 GB 2024/11/19-15:17:54.138423 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004589.sst): second pass time = 0.00's,133992737.624'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 576.417'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3909.325'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.623'MB/sec write SST data time = 0.05's, 1069.585'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 133992737.624'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463425 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 117.86 usrkeys = 463425 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6615 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5693 value = 0.9911 dict = 0.00 all = 1.2091 } Zip/UnZip{ index = 0.0946 value = 1.0090 dict = 0.00 all = 0.8270 } ---------------------------- total value len = 2.855013 GB avg = 0.368 KB (by entry num) total key len = 0.190692 GB avg = 0.025 KB total ukey len = 0.128578 GB avg = 0.017 KB total ukey num = 0.007764271 Billion total entry num = 0.007764271 Billion write speed all = 16.976439570 MB/sec (with seq num) write speed all = 16.630221736 MB/sec (without seq num) 2024/11/19-15:17:54.142002 139851137550080 [compaction_job.cc:1948] [default] [JOB 74] Generated table #4589: 463425 keys, 55501528 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:54.142056 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000674142033, "cf_name": "default", "job": 74, "event": "table_file_creation", "file_number": 4589, "file_size": 55501528, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54619136, "index_size": 876928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12975900, "raw_average_key_size": 28, "raw_value_size": 54133066, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463425, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000673, "oldest_key_time": 0, "file_creation_time": 1732000673, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4589, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:17:54.158815 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004590.sst) /storage/mytopling/..//tmp/Topling-82038-tE2YDr 2024/11/19-15:17:54.270272 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004590.sst: sumWaitingMem = 0.005 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0047 GB, waited 0.000 sec, Key+Value = 0.052 GB 2024/11/19-15:17:54.297380 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004590.sst): index pass time = 0.03's, 213.718'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 361620 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0058 GB zip-key = 0.0007 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:17:54.346620 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-00 2024/11/19-15:17:54.346714 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00081/att-00/compact.done) = No such file or directory: 5.023 sec on http probe failed: 3, retry 2024/11/19-15:17:54.348225 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004590.sst): first pass time = 0.19's, 276.492'MB/sec 2024/11/19-15:17:54.348273 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004590.sst): old prealloc_size = 73819750, real_size = 43395408 2024/11/19-15:17:54.348461 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004590.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 361620 prefix = 4 raw-key = 0.0058 GB zip-key = 0.0007 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0422 GB zip-val = 0.0426 GB avg-val = 116.81 avg-zval = 117.86 2024/11/19-15:17:54.348518 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004590.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0009 GB, waited 0.078 sec, Key+Value = 0.052 GB 2024/11/19-15:17:54.420874 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004590.sst): second pass time = 0.00's,88000666.667'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 596.662'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3789.087'MB/sec (index lex order gen) rebuild zvType time = 0.03's, 2.641'MB/sec write SST data time = 0.04's, 1134.302'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.042'GB zip my value throughput = 88000666.667'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 361620 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 117.86 usrkeys = 361620 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 5163 UnZipSize{ index = 0.0072 GB value = 0.0422 GB dict = 0.00 MB all = 0.0524 GB } __ZipSize{ index = 0.0007 GB value = 0.0426 GB dict = 0.00 MB all = 0.0433 GB } UnZip/Zip{ index = 10.5653 value = 0.9911 dict = 0.00 all = 1.2091 } Zip/UnZip{ index = 0.0946 value = 1.0090 dict = 0.00 all = 0.8271 } ---------------------------- total value len = 2.897253 GB avg = 0.357 KB (by entry num) total key len = 0.200817 GB avg = 0.025 KB total ukey len = 0.135810 GB avg = 0.017 KB total ukey num = 0.008125891 Billion total entry num = 0.008125891 Billion write speed all = 17.241175644 MB/sec (with seq num) write speed all = 16.879402336 MB/sec (without seq num) 2024/11/19-15:17:54.423423 139851137550080 [compaction_job.cc:1948] [default] [JOB 74] Generated table #4590: 361620 keys, 43309400 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:54.423474 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000674423451, "cf_name": "default", "job": 74, "event": "table_file_creation", "file_number": 4590, "file_size": 43309400, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 42620288, "index_size": 684544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 10125360, "raw_average_key_size": 28, "raw_value_size": 42240320, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 361620, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000674, "oldest_key_time": 0, "file_creation_time": 1732000674, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4590, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:17:54.433850 139851137550080 [compaction_job.cc:703] job-00074: subcompact[0], size: 0.098811 G, files: /storage/mytopling/.rocksdb [004589,004590] 2024/11/19-15:17:54.435071 139851137550080 (Original Log Time 2024/11/19-15:17:53.782012) EVENT_LOG_v1 {"time_micros": 1732000673781982, "job": 74, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4503, 4506, 4474, 4479], "score": 4.39084, "input_data_size": 118885972, "oldest_snapshot_seqno": -1} 2024/11/19-15:17:54.435081 139851137550080 (Original Log Time 2024/11/19-15:17:54.434687) [compaction_job.cc:2022] [default] [JOB 74] Compacted 4@1 files to L2 => 98810928 bytes 2024/11/19-15:17:54.435084 139851137550080 (Original Log Time 2024/11/19-15:17:54.434960) [compaction_job.cc:1224] [default] compacted to: files[3 73 19 44 0 0 0] max score 3.27 (71 files need compaction), MB/sec: 182.4 rd, 151.6 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(113.4, 0.0 +0.0 blob) out(94.2 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 825045, records dropped: 0 output_compression: Snappy 2024/11/19-15:17:54.435099 139851137550080 (Original Log Time 2024/11/19-15:17:54.434987) EVENT_LOG_v1 {"time_micros": 1732000674434974, "cf": "default", "job": 74, "event": "compaction_finished", "compaction_time_micros": 651849, "compaction_time_cpu_micros": 648424, "output_level": 2, "num_output_files": 2, "total_output_size": 98810928, "num_input_records": 825045, "num_output_records": 825045, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 73, 19, 44, 0, 0, 0]} 2024/11/19-15:17:54.435168 139851137550080 [compaction_job.cc:2437] [default] [JOB 82] Compacting 4@1 files to L2, score 3.27, subcompactions 1 : 1 2024/11/19-15:17:54.435176 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 80 Base level 1, inputs: [4518(31MB) 4519(31MB) 4520(31MB) 4521(31MB)] 2024/11/19-15:17:54.450028 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-02 2024/11/19-15:17:54.450091 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00073/att-02/compact.done) = No such file or directory: 11.221 sec on http probe failed: 5, giveup 2024/11/19-15:17:54.450108 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00073/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.818 sec, real 11.513 (submit 279.720 ms wait 11.221 s), input 127.818 MiB, send shutdown 2024/11/19-15:17:54.458499 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00073/att-02 2024/11/19-15:17:54.458693 139851158521600 [dcompact_etcd.cc:1773] job-00073/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:54.460568 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00073) = success 2024/11/19-15:17:54.460575 139851158521600 [dcompact_etcd.cc:1773] job-00073/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:17:54.460592 139851158521600 (Original Log Time 2024/11/19-15:17:19.892994) EVENT_LOG_v1 {"time_micros": 1732000639892989, "job": 73, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4492, 4494, 4496, 4499], "score": 4.64048, "input_data_size": 133300308, "oldest_snapshot_seqno": -1} 2024/11/19-15:17:54.460600 139851158521600 [compaction_job.cc:2437] [default] [JOB 73] Compacting 4@1 files to L2, score 4.64, subcompactions 1 : 1 2024/11/19-15:17:54.460610 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 71 Base level 1, inputs: [4492(31MB) 4494(31MB) 4496(31MB) 4499(31MB)] 2024/11/19-15:17:54.460828 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004591.sst) /storage/mytopling/..//tmp/Topling-82038-luhYcN 2024/11/19-15:17:54.470711 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-01 2024/11/19-15:17:54.470776 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00078/att-01/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:17:54.624367 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004591.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:17:54.672133 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004591.sst): index pass time = 0.05's, 155.356'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463360 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:17:54.700773 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-02 2024/11/19-15:17:54.700886 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00075/att-02/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:17:54.708795 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-01 2024/11/19-15:17:54.708877 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00079/att-01/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:17:54.716715 139851137550080 [dcompact_etcd.cc:1163] job-00082/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 126.957 MiB 2024/11/19-15:17:54.725143 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004591.sst): first pass time = 0.26's, 253.910'MB/sec 2024/11/19-15:17:54.725189 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004591.sst): old prealloc_size = 73819750, real_size = 55606544 2024/11/19-15:17:54.725424 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004591.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463360 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.83 avg-zval = 117.86 2024/11/19-15:17:54.725486 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004591.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.101 sec, Key+Value = 0.067 GB 2024/11/19-15:17:54.819437 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004591.sst): second pass time = 0.00's,145915902.965'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 588.941'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3782.262'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 2.554'MB/sec write SST data time = 0.05's, 1140.753'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 145915902.965'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463360 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.83 avg-zval = 117.86 usrkeys = 463360 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.83 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6615 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5701 value = 0.9912 dict = 0.00 all = 1.2093 } Zip/UnZip{ index = 0.0946 value = 1.0088 dict = 0.00 all = 0.8270 } ---------------------------- total value len = 2.951388 GB avg = 0.344 KB (by entry num) total key len = 0.213791 GB avg = 0.025 KB total ukey len = 0.145077 GB avg = 0.017 KB total ukey num = 0.008589251 Billion total entry num = 0.008589251 Billion write speed all = 17.575661824 MB/sec (with seq num) write speed all = 17.194105546 MB/sec (without seq num) 2024/11/19-15:17:54.822841 139851158521600 [compaction_job.cc:1948] [default] [JOB 73] Generated table #4591: 463360 keys, 55495960 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:54.822891 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000674822868, "cf_name": "default", "job": 73, "event": "table_file_creation", "file_number": 4591, "file_size": 55495960, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54613760, "index_size": 876736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12974080, "raw_average_key_size": 28, "raw_value_size": 54134800, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463360, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000674, "oldest_key_time": 0, "file_creation_time": 1732000674, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4591, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:17:54.836837 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004592.sst) /storage/mytopling/..//tmp/Topling-82038-QGycHO 2024/11/19-15:17:54.980601 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004592.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:17:55.016100 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004592.sst): index pass time = 0.04's, 208.526'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462188 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:17:55.080837 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004592.sst): first pass time = 0.24's, 274.274'MB/sec 2024/11/19-15:17:55.080890 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004592.sst): old prealloc_size = 73819750, real_size = 55466416 2024/11/19-15:17:55.081129 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004592.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462188 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.79 avg-zval = 117.85 2024/11/19-15:17:55.081192 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004592.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.101 sec, Key+Value = 0.067 GB 2024/11/19-15:17:55.174243 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004592.sst): second pass time = 0.00's,136305444.444'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 593.062'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3714.536'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.620'MB/sec write SST data time = 0.05's, 1129.766'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 136305444.444'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462188 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 117.85 usrkeys = 462188 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6597 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0554 GB } UnZip/Zip{ index = 10.5012 value = 0.9909 dict = 0.00 all = 1.2089 } Zip/UnZip{ index = 0.0952 value = 1.0091 dict = 0.00 all = 0.8272 } ---------------------------- total value len = 3.005365 GB avg = 0.332 KB (by entry num) total key len = 0.226733 GB avg = 0.025 KB total ukey len = 0.154321 GB avg = 0.017 KB total ukey num = 0.009051439 Billion total entry num = 0.009051439 Billion write speed all = 17.911956667 MB/sec (with seq num) write speed all = 17.510659455 MB/sec (without seq num) 2024/11/19-15:17:55.177599 139851158521600 [compaction_job.cc:1948] [default] [JOB 73] Generated table #4592: 462188 keys, 55356184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:55.177650 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000675177627, "cf_name": "default", "job": 73, "event": "table_file_creation", "file_number": 4592, "file_size": 55356184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54470432, "index_size": 880256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12941264, "raw_average_key_size": 28, "raw_value_size": 53976956, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462188, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000674, "oldest_key_time": 0, "file_creation_time": 1732000674, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4592, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:17:55.178865 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-02 2024/11/19-15:17:55.178956 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00076/att-02/compact.done) = No such file or directory: 9.221 sec on http probe failed: 4, retry 2024/11/19-15:17:55.190874 139851158521600 [compaction_job.cc:703] job-00073: subcompact[0], size: 0.110852 G, files: /storage/mytopling/.rocksdb [004591,004592] 2024/11/19-15:17:55.192200 139851158521600 (Original Log Time 2024/11/19-15:17:54.460636) EVENT_LOG_v1 {"time_micros": 1732000674460620, "job": 73, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4492, 4494, 4496, 4499], "score": 4.64048, "input_data_size": 133300308, "oldest_snapshot_seqno": -1} 2024/11/19-15:17:55.192209 139851158521600 (Original Log Time 2024/11/19-15:17:55.191836) [compaction_job.cc:2022] [default] [JOB 73] Compacted 4@1 files to L2 => 110852144 bytes 2024/11/19-15:17:55.192213 139851158521600 (Original Log Time 2024/11/19-15:17:55.192096) [compaction_job.cc:1224] [default] compacted to: files[3 69 21 44 0 0 0] max score 3.02 (69 files need compaction), MB/sec: 182.5 rd, 151.8 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925548, records dropped: 0 output_compression: Snappy 2024/11/19-15:17:55.192229 139851158521600 (Original Log Time 2024/11/19-15:17:55.192120) EVENT_LOG_v1 {"time_micros": 1732000675192108, "cf": "default", "job": 73, "event": "compaction_finished", "compaction_time_micros": 730250, "compaction_time_cpu_micros": 725981, "output_level": 2, "num_output_files": 2, "total_output_size": 110852144, "num_input_records": 925548, "num_output_records": 925548, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 69, 21, 44, 0, 0, 0]} 2024/11/19-15:17:55.198762 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004457.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:55.198786 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000675198783, "job": 73, "event": "table_file_deletion", "file_number": 4457} 2024/11/19-15:17:55.206309 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004498.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:17:55.206332 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000675206329, "job": 73, "event": "table_file_deletion", "file_number": 4498} 2024/11/19-15:17:55.206468 139851158521600 [compaction_job.cc:2437] [default] [JOB 83] Compacting 4@1 files to L2, score 3.02, subcompactions 1 : 1 2024/11/19-15:17:55.206477 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 81 Base level 1, inputs: [4522(31MB) 4523(31MB) 4524(31MB) 4525(31MB)] 2024/11/19-15:17:55.493300 139851158521600 [dcompact_etcd.cc:1163] job-00083/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.286 sec, input = 126.908 MiB 2024/11/19-15:17:55.895654 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-00 2024/11/19-15:17:55.895732 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00081/att-00/compact.done) = No such file or directory: 6.573 sec on http probe failed: 4, retry 2024/11/19-15:17:56.480253 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-01 2024/11/19-15:17:56.480357 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00078/att-01/compact.done) = No such file or directory: 7.209 sec on http probe failed: 3, retry 2024/11/19-15:17:56.706753 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-02 2024/11/19-15:17:56.706835 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00075/att-02/compact.done) = No such file or directory: 9.018 sec on http probe failed: 3, retry 2024/11/19-15:17:56.717240 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-01 2024/11/19-15:17:56.717297 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00079/att-01/compact.done) = No such file or directory: 7.214 sec on http probe failed: 3, retry 2024/11/19-15:17:57.187104 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-02 2024/11/19-15:17:57.187173 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00076/att-02/compact.done) = No such file or directory: 11.226 sec on http probe failed: 5, giveup 2024/11/19-15:17:57.187191 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00076/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.821 sec, real 11.527 (submit 292.615 ms wait 11.226 s), input 127.821 MiB, send shutdown 2024/11/19-15:17:57.195211 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00076/att-02 2024/11/19-15:17:57.195426 139851127064320 [dcompact_etcd.cc:1773] job-00076/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:57.197928 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00076) = success 2024/11/19-15:17:57.197938 139851127064320 [dcompact_etcd.cc:1773] job-00076/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:17:57.197958 139851127064320 (Original Log Time 2024/11/19-15:17:22.472112) EVENT_LOG_v1 {"time_micros": 1732000642472104, "job": 76, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4485, 4489, 4493, 4495], "score": 4.1683, "input_data_size": 133300052, "oldest_snapshot_seqno": 115813221} 2024/11/19-15:17:57.197965 139851127064320 [compaction_job.cc:2437] [default] [JOB 76] Compacting 4@1 files to L2, score 4.17, subcompactions 1 : 1 2024/11/19-15:17:57.197973 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 74 Base level 1, inputs: [4485(31MB) 4489(31MB) 4493(31MB) 4495(31MB)] 2024/11/19-15:17:57.198200 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004593.sst) /storage/mytopling/..//tmp/Topling-82038-otdupJ 2024/11/19-15:17:57.348927 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004593.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:17:57.383427 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004593.sst): index pass time = 0.03's, 215.194'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463517 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:17:57.444608 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-00 2024/11/19-15:17:57.444713 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00081/att-00/compact.done) = No such file or directory: 8.122 sec on http probe failed: 5, giveup 2024/11/19-15:17:57.444738 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00081/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 77.023 sec, real 8.448 (submit 318.378 ms wait 8.122 s), input 77.023 MiB, send shutdown 2024/11/19-15:17:57.450280 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004593.sst): first pass time = 0.25's, 266.232'MB/sec 2024/11/19-15:17:57.450324 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004593.sst): old prealloc_size = 73819750, real_size = 55619376 2024/11/19-15:17:57.450547 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004593.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463517 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.78 avg-zval = 117.85 2024/11/19-15:17:57.450607 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004593.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.102 sec, Key+Value = 0.067 GB 2024/11/19-15:17:57.452787 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-00 2024/11/19-15:17:57.452999 139851169007360 [dcompact_etcd.cc:1773] job-00081/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:17:57.544167 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004593.sst): second pass time = 0.00's,233320681.034'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 591.626'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 4003.672'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.615'MB/sec write SST data time = 0.05's, 1126.157'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 233320681.034'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463517 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 117.85 usrkeys = 463517 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6615 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5737 value = 0.9909 dict = 0.00 all = 1.2090 } Zip/UnZip{ index = 0.0946 value = 1.0092 dict = 0.00 all = 0.8271 } ---------------------------- total value len = 3.059496 GB avg = 0.322 KB (by entry num) total key len = 0.239711 GB avg = 0.025 KB total ukey len = 0.163592 GB avg = 0.017 KB total ukey num = 0.009514956 Billion total entry num = 0.009514956 Billion write speed all = 18.046842432 MB/sec (with seq num) write speed all = 17.630463769 MB/sec (without seq num) 2024/11/19-15:17:57.547595 139851127064320 [compaction_job.cc:1948] [default] [JOB 76] Generated table #4593: 463517 keys, 55508824 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:57.547649 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000677547624, "cf_name": "default", "job": 76, "event": "table_file_creation", "file_number": 4593, "file_size": 55508824, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54626592, "index_size": 876736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12978476, "raw_average_key_size": 28, "raw_value_size": 54130398, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463517, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000677, "oldest_key_time": 0, "file_creation_time": 1732000677, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4593, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:17:57.561295 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004594.sst) /storage/mytopling/..//tmp/Topling-82038-VImUVA 2024/11/19-15:17:57.710827 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004594.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:17:57.729403 139851169007360 [dcompact_etcd.cc:1163] job-00081/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.276 sec, input = 77.023 MiB 2024/11/19-15:17:57.758910 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004594.sst): index pass time = 0.05's, 153.931'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462174 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:17:57.822748 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004594.sst): first pass time = 0.26's, 255.977'MB/sec 2024/11/19-15:17:57.822800 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004594.sst): old prealloc_size = 73819750, real_size = 55459968 2024/11/19-15:17:57.823040 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004594.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462174 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.80 avg-zval = 117.86 2024/11/19-15:17:57.823107 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004594.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.112 sec, Key+Value = 0.067 GB 2024/11/19-15:17:57.899269 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-00 2024/11/19-15:17:57.899375 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00082/att-00/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:17:57.916541 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004594.sst): second pass time = 0.00's,184864863.014'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 590.530'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3663.711'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.633'MB/sec write SST data time = 0.05's, 1115.797'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 184864863.014'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462174 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.80 avg-zval = 117.86 usrkeys = 462174 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.80 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6597 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0553 GB } UnZip/Zip{ index = 10.5739 value = 0.9910 dict = 0.00 all = 1.2091 } Zip/UnZip{ index = 0.0946 value = 1.0091 dict = 0.00 all = 0.8271 } ---------------------------- total value len = 3.113476 GB avg = 0.312 KB (by entry num) total key len = 0.252652 GB avg = 0.025 KB total ukey len = 0.172835 GB avg = 0.017 KB total ukey num = 0.009977130 Billion total entry num = 0.009977130 Billion write speed all = 18.375476586 MB/sec (with seq num) write speed all = 17.939760511 MB/sec (without seq num) 2024/11/19-15:17:57.919898 139851127064320 [compaction_job.cc:1948] [default] [JOB 76] Generated table #4594: 462174 keys, 55349720 bytes (need compaction), temperature: kUnknown 2024/11/19-15:17:57.919960 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000677919937, "cf_name": "default", "job": 76, "event": "table_file_creation", "file_number": 4594, "file_size": 55349720, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54470064, "index_size": 874176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12940872, "raw_average_key_size": 28, "raw_value_size": 53980540, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462174, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000677, "oldest_key_time": 0, "file_creation_time": 1732000677, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4594, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:17:57.933460 139851127064320 [compaction_job.cc:703] job-00076: subcompact[0], size: 0.110859 G, files: /storage/mytopling/.rocksdb [004593,004594] 2024/11/19-15:17:57.934719 139851127064320 (Original Log Time 2024/11/19-15:17:57.198001) EVENT_LOG_v1 {"time_micros": 1732000677197985, "job": 76, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4485, 4489, 4493, 4495], "score": 4.1683, "input_data_size": 133300052, "oldest_snapshot_seqno": 115813221} 2024/11/19-15:17:57.934728 139851127064320 (Original Log Time 2024/11/19-15:17:57.934360) [compaction_job.cc:2022] [default] [JOB 76] Compacted 4@1 files to L2 => 110858544 bytes 2024/11/19-15:17:57.934731 139851127064320 (Original Log Time 2024/11/19-15:17:57.934607) [compaction_job.cc:1224] [default] compacted to: files[3 65 23 44 0 0 0] max score 2.77 (67 files need compaction), MB/sec: 181.2 rd, 150.7 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925691, records dropped: 0 output_compression: Snappy 2024/11/19-15:17:57.934745 139851127064320 (Original Log Time 2024/11/19-15:17:57.934631) EVENT_LOG_v1 {"time_micros": 1732000677934619, "cf": "default", "job": 76, "event": "compaction_finished", "compaction_time_micros": 735472, "compaction_time_cpu_micros": 732070, "output_level": 2, "num_output_files": 2, "total_output_size": 110858544, "num_input_records": 925691, "num_output_records": 925691, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 65, 23, 44, 0, 0, 0]} 2024/11/19-15:17:57.934815 139851127064320 [compaction_job.cc:2437] [default] [JOB 84] Compacting 4@1 files to L2, score 2.77, subcompactions 1 : 1 2024/11/19-15:17:57.934823 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 82 Base level 1, inputs: [4526(31MB) 4527(31MB) 4528(31MB) 4529(31MB)] 2024/11/19-15:17:58.201568 139851127064320 [dcompact_etcd.cc:1163] job-00084/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.266 sec, input = 126.878 MiB 2024/11/19-15:17:58.488602 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-01 2024/11/19-15:17:58.488683 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00078/att-01/compact.done) = No such file or directory: 9.219 sec on http probe failed: 4, retry 2024/11/19-15:17:58.674306 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-00 2024/11/19-15:17:58.674399 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00083/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:17:58.716262 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-02 2024/11/19-15:17:58.716339 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00075/att-02/compact.done) = No such file or directory: 11.024 sec on http probe failed: 4, retry 2024/11/19-15:17:58.728012 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-01 2024/11/19-15:17:58.728071 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00079/att-01/compact.done) = No such file or directory: 9.223 sec on http probe failed: 4, retry 2024/11/19-15:17:59.663192 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-01 2024/11/19-15:17:59.663295 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00081/att-01/compact.done) = No such file or directory: 1.926 sec on http probe failed: 1, retry 2024/11/19-15:17:59.908245 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-00 2024/11/19-15:17:59.908344 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00082/att-00/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:18:00.498435 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-01 2024/11/19-15:18:00.498515 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00078/att-01/compact.done) = No such file or directory: 11.227 sec on http probe failed: 5, giveup 2024/11/19-15:18:00.498535 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00078/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.819 sec, real 11.485 (submit 247.731 ms wait 11.227 s), input 127.819 MiB, send shutdown 2024/11/19-15:18:00.507496 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-01 2024/11/19-15:18:00.507713 139851179493120 [dcompact_etcd.cc:1773] job-00078/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:00.683066 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-00 2024/11/19-15:18:00.683153 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00083/att-00/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:18:00.725023 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-02 2024/11/19-15:18:00.725119 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00075/att-02/compact.done) = No such file or directory: 13.033 sec on http probe failed: 5, giveup 2024/11/19-15:18:00.725146 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00075/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 248.724 sec, real 13.331 (submit 289.165 ms wait 13.033 s), input 248.724 MiB, send shutdown 2024/11/19-15:18:00.730826 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00075/att-02 2024/11/19-15:18:00.731052 139851148035840 [dcompact_etcd.cc:1773] job-00075/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:00.732929 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00075) = success 2024/11/19-15:18:00.732939 139851148035840 [dcompact_etcd.cc:1773] job-00075/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:18:00.732959 139851148035840 (Original Log Time 2024/11/19-15:17:20.695002) EVENT_LOG_v1 {"time_micros": 1732000640694993, "job": 75, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L2": [4501, 4507, 4502, 4508], "score": -1, "input_data_size": 256820288, "oldest_snapshot_seqno": -1} 2024/11/19-15:18:00.732967 139851148035840 [compaction_job.cc:2437] [default] [JOB 75] Compacting 4@2 files to L3, score -1.00, subcompactions 1 : 1 2024/11/19-15:18:00.732974 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 73 Base level 2, inputs: [4501(63MB) 4507(61MB) 4502(63MB) 4508(56MB)] 2024/11/19-15:18:00.737081 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-01 2024/11/19-15:18:00.737148 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00079/att-01/compact.done) = No such file or directory: 11.233 sec on http probe failed: 5, giveup 2024/11/19-15:18:00.737170 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00079/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.541 (submit 298.240 ms wait 11.233 s), input 127.820 MiB, send shutdown 2024/11/19-15:18:00.745719 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-01 2024/11/19-15:18:00.745950 139850978166528 [dcompact_etcd.cc:1773] job-00079/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:00.755879 139851148035840 [compaction_job.cc:703] job-00075: subcompact[0], size: 0.000000 G, files: /storage/mytopling/.rocksdb [] 2024/11/19-15:18:00.756290 139851148035840 (Original Log Time 2024/11/19-15:18:00.733007) EVENT_LOG_v1 {"time_micros": 1732000680732987, "job": 75, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L2": [4501, 4507, 4502, 4508], "score": -1, "input_data_size": 256820288, "oldest_snapshot_seqno": -1} 2024/11/19-15:18:00.756300 139851148035840 (Original Log Time 2024/11/19-15:18:00.755942) [compaction_job.cc:2022] [default] [JOB 75] Compacted 4@2 files to L3 => 0 bytes 2024/11/19-15:18:00.756303 139851148035840 (Original Log Time 2024/11/19-15:18:00.756180) [compaction_job.cc:1224] [default] compacted to: files[3 65 19 44 0 0 0] max score 2.52 (63 files need compaction), MB/sec: 11222.2 rd, 0.0 wr, level 3, files in(4, 0) out(0 +0 blob) MB in(244.9, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.0) write-amplify(0.0) OK, records in: 251502, records dropped: 251502 output_compression: Snappy 2024/11/19-15:18:00.756317 139851148035840 (Original Log Time 2024/11/19-15:18:00.756206) EVENT_LOG_v1 {"time_micros": 1732000680756193, "cf": "default", "job": 75, "event": "compaction_finished", "compaction_time_micros": 22885, "compaction_time_cpu_micros": 22789, "output_level": 3, "num_output_files": 0, "total_output_size": 0, "num_input_records": 251502, "num_output_records": 0, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 65, 19, 44, 0, 0, 0]} 2024/11/19-15:18:00.756503 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004473.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:00.756519 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000680756516, "job": 75, "event": "table_file_deletion", "file_number": 4473} 2024/11/19-15:18:00.756657 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004475.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:00.756667 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000680756665, "job": 75, "event": "table_file_deletion", "file_number": 4475} 2024/11/19-15:18:00.761933 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004476.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:00.761961 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000680761958, "job": 75, "event": "table_file_deletion", "file_number": 4476} 2024/11/19-15:18:00.766503 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004477.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:00.766530 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000680766527, "job": 75, "event": "table_file_deletion", "file_number": 4477} 2024/11/19-15:18:00.770896 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004478.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:00.770918 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000680770915, "job": 75, "event": "table_file_deletion", "file_number": 4478} 2024/11/19-15:18:00.775030 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004480.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:00.775051 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000680775048, "job": 75, "event": "table_file_deletion", "file_number": 4480} 2024/11/19-15:18:00.779032 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004482.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:00.779053 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000680779051, "job": 75, "event": "table_file_deletion", "file_number": 4482} 2024/11/19-15:18:00.783001 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004484.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:00.783021 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000680783019, "job": 75, "event": "table_file_deletion", "file_number": 4484} 2024/11/19-15:18:00.783051 139834427442944 (Original Log Time 2024/11/19-15:16:57.911111) [db_impl_compaction_flush.cc:2043] [default] Manual compaction starting 2024/11/19-15:18:00.783079 139834427442944 (Original Log Time 2024/11/19-15:18:00.783075) [db_impl_compaction_flush.cc:2043] [default] Manual compaction starting 2024/11/19-15:18:00.783214 139851148035840 [compaction_job.cc:2437] [default] [JOB 85] Compacting 4@1 files to L2, score 2.52, subcompactions 1 : 1 2024/11/19-15:18:00.783223 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 83 Base level 1, inputs: [4532(31MB) 4533(31MB) 4534(31MB) 4535(31MB)] 2024/11/19-15:18:00.788580 139851179493120 [dcompact_etcd.cc:1163] job-00078/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 127.819 MiB 2024/11/19-15:18:01.052233 139850978166528 [dcompact_etcd.cc:1163] job-00079/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.306 sec, input = 127.820 MiB 2024/11/19-15:18:01.073973 139851148035840 [dcompact_etcd.cc:1163] job-00085/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.290 sec, input = 126.905 MiB 2024/11/19-15:18:01.212527 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-01 2024/11/19-15:18:01.212662 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00081/att-01/compact.done) = No such file or directory: 3.474 sec on http probe failed: 2, retry 2024/11/19-15:18:01.379973 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-00 2024/11/19-15:18:01.380061 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00084/att-00/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:18:01.913906 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-00 2024/11/19-15:18:01.913991 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00082/att-00/compact.done) = No such file or directory: 7.192 sec on http probe failed: 3, retry 2024/11/19-15:18:02.691464 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-00 2024/11/19-15:18:02.691558 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00083/att-00/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:18:02.762634 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-01 2024/11/19-15:18:02.762729 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00081/att-01/compact.done) = No such file or directory: 5.024 sec on http probe failed: 3, retry 2024/11/19-15:18:03.388965 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-00 2024/11/19-15:18:03.389037 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00084/att-00/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:18:03.922668 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-00 2024/11/19-15:18:03.922745 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00082/att-00/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:18:03.989270 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-02 2024/11/19-15:18:03.989338 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00078/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:18:04.257267 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-02 2024/11/19-15:18:04.257355 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00079/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:18:04.257540 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-00 2024/11/19-15:18:04.257610 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00085/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:18:04.309013 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-01 2024/11/19-15:18:04.309139 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00081/att-01/compact.done) = No such file or directory: 6.574 sec on http probe failed: 4, retry 2024/11/19-15:18:04.696953 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-00 2024/11/19-15:18:04.697047 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00083/att-00/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:18:05.398897 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-00 2024/11/19-15:18:05.398982 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00084/att-00/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:18:05.536836 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4595. Immutable memtables: 0. 2024/11/19-15:18:05.537372 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 93] Syncing log #4585 2024/11/19-15:18:05.537460 139850967680768 (Original Log Time 2024/11/19-15:18:05.537159) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:18:05.537472 139850967680768 [flush_job.cc:879] [default] [JOB 93] Flushing memtable with next log file: 4595 2024/11/19-15:18:05.537493 139850967680768 EVENT_LOG_v1 {"time_micros": 1732000685537488, "job": 93, "event": "flush_started", "num_memtables": 1, "num_entries": 1416430, "num_deletes": 0, "total_data_size": 459847593, "memory_usage": 536608136, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:18:05.537497 139850967680768 [flush_job.cc:910] [default] [JOB 93] Level-0 flush table #4596: started 2024/11/19-15:18:05.537810 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000009.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004596.sst in ConvertToSST, mem_size = 511.749 M, time = 304.744 us 2024/11/19-15:18:05.859264 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-01 2024/11/19-15:18:05.859473 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00081/att-01/compact.done) = No such file or directory: 8.120 sec on http probe failed: 5, giveup 2024/11/19-15:18:05.859520 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00081/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 77.023 sec, real 8.406 (submit 276.328 ms wait 8.120 s), input 77.023 MiB, send shutdown 2024/11/19-15:18:05.865888 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-01 2024/11/19-15:18:05.869167 139851169007360 [dcompact_etcd.cc:1773] job-00081/att-01: CleanFiles 0.003 sec : 0.000 2024/11/19-15:18:05.930839 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-00 2024/11/19-15:18:05.930962 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00082/att-00/compact.done) = No such file or directory: 11.206 sec on http probe failed: 5, giveup 2024/11/19-15:18:05.930995 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00082/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.957 sec, real 11.495 (submit 281.071 ms wait 11.206 s), input 126.957 MiB, send shutdown 2024/11/19-15:18:05.936252 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-00 2024/11/19-15:18:05.937055 139851137550080 [dcompact_etcd.cc:1773] job-00082/att-00: CleanFiles 0.001 sec : 0.000 2024/11/19-15:18:05.994642 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-02 2024/11/19-15:18:05.994747 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00078/att-02/compact.done) = No such file or directory: 5.201 sec on http probe failed: 2, retry 2024/11/19-15:18:06.047163 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004596.sst): fsize = 511.751 M, time(ms): open: 0.032, seek: 0.013, finish: 0.034, meta: 0.012, Flush: 0.022, sync: 509.213, close: 0.011, all: 509.337 2024/11/19-15:18:06.048331 139850967680768 [flush_job.cc:1083] [default] [JOB 93] Flush lasted 510871 microseconds, and 431394 cpu microseconds. 2024/11/19-15:18:06.068282 139850967680768 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:18:06.068353 139850967680768 (Original Log Time 2024/11/19-15:18:06.047207) [flush_job.cc:1029] [default] [JOB 93] Level-0 flush table #4596: 536609562 bytes OK (needs compaction) 2024/11/19-15:18:06.068359 139850967680768 (Original Log Time 2024/11/19-15:18:06.048348) [memtable_list.cc:560] [default] Level-0 commit table #4596 started 2024/11/19-15:18:06.068362 139850967680768 (Original Log Time 2024/11/19-15:18:06.068250) [memtable_list.cc:759] [default] Level-0 commit table #4596: memtable #1 done 2024/11/19-15:18:06.068365 139850967680768 (Original Log Time 2024/11/19-15:18:06.068276) EVENT_LOG_v1 {"time_micros": 1732000686068270, "job": 93, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [4, 65, 18, 0, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:18:06.068368 139850967680768 (Original Log Time 2024/11/19-15:18:06.068324) [db_impl_compaction_flush.cc:310] [default] Level summary: files[4 65 18 0 0 0 0] max score 2.27 (41 files need compaction) 2024/11/19-15:18:06.195214 139851169007360 [dcompact_etcd.cc:1163] job-00081/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.326 sec, input = 77.023 MiB 2024/11/19-15:18:06.256009 139851137550080 [dcompact_etcd.cc:1163] job-00082/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.319 sec, input = 126.957 MiB 2024/11/19-15:18:06.265992 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-02 2024/11/19-15:18:06.266079 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00079/att-02/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:18:06.266139 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-00 2024/11/19-15:18:06.266198 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00085/att-00/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:18:06.706636 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-00 2024/11/19-15:18:06.706708 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00083/att-00/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:18:06.706727 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00083/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.908 sec, real 11.500 (submit 286.341 ms wait 11.204 s), input 126.908 MiB, send shutdown 2024/11/19-15:18:06.716056 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-00 2024/11/19-15:18:06.716282 139851158521600 [dcompact_etcd.cc:1773] job-00083/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:07.019058 139851158521600 [dcompact_etcd.cc:1163] job-00083/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.303 sec, input = 126.908 MiB 2024/11/19-15:18:07.408322 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-00 2024/11/19-15:18:07.408398 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00084/att-00/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:18:08.004151 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-02 2024/11/19-15:18:08.004244 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00078/att-02/compact.done) = No such file or directory: 7.206 sec on http probe failed: 3, retry 2024/11/19-15:18:08.130880 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-02 2024/11/19-15:18:08.130996 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00081/att-02/compact.done) = No such file or directory: 1.926 sec on http probe failed: 1, retry 2024/11/19-15:18:08.271429 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-00 2024/11/19-15:18:08.271546 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00085/att-00/compact.done) = No such file or directory: 7.192 sec on http probe failed: 3, retry 2024/11/19-15:18:08.274655 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-02 2024/11/19-15:18:08.274731 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00079/att-02/compact.done) = No such file or directory: 7.214 sec on http probe failed: 3, retry 2024/11/19-15:18:09.413263 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-00 2024/11/19-15:18:09.413356 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00084/att-00/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:18:09.413380 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00084/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.878 sec, real 11.478 (submit 266.258 ms wait 11.207 s), input 126.878 MiB, send shutdown 2024/11/19-15:18:09.418782 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-00 2024/11/19-15:18:09.419005 139851127064320 [dcompact_etcd.cc:1773] job-00084/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:09.438442 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-01 2024/11/19-15:18:09.438513 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00082/att-01/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:18:09.680705 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-02 2024/11/19-15:18:09.680799 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00081/att-02/compact.done) = No such file or directory: 3.476 sec on http probe failed: 2, retry 2024/11/19-15:18:09.702437 139851127064320 [dcompact_etcd.cc:1163] job-00084/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.283 sec, input = 126.878 MiB 2024/11/19-15:18:10.014345 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-02 2024/11/19-15:18:10.014442 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00078/att-02/compact.done) = No such file or directory: 9.216 sec on http probe failed: 4, retry 2024/11/19-15:18:10.197942 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-01 2024/11/19-15:18:10.198042 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00083/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:18:10.278807 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-00 2024/11/19-15:18:10.278906 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00085/att-00/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:18:10.282276 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-02 2024/11/19-15:18:10.282336 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00079/att-02/compact.done) = No such file or directory: 9.223 sec on http probe failed: 4, retry 2024/11/19-15:18:11.230384 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-02 2024/11/19-15:18:11.230501 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00081/att-02/compact.done) = No such file or directory: 5.026 sec on http probe failed: 3, retry 2024/11/19-15:18:11.443789 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-01 2024/11/19-15:18:11.443872 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00082/att-01/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:18:12.023990 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-02 2024/11/19-15:18:12.024088 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00078/att-02/compact.done) = No such file or directory: 11.226 sec on http probe failed: 5, giveup 2024/11/19-15:18:12.024114 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00078/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.819 sec, real 11.516 (submit 280.790 ms wait 11.226 s), input 127.819 MiB, send shutdown 2024/11/19-15:18:12.030627 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00078/att-02 2024/11/19-15:18:12.030884 139851179493120 [dcompact_etcd.cc:1773] job-00078/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:12.032823 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00078) = success 2024/11/19-15:18:12.032838 139851179493120 [dcompact_etcd.cc:1773] job-00078/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:18:12.032861 139851179493120 (Original Log Time 2024/11/19-15:17:37.483498) EVENT_LOG_v1 {"time_micros": 1732000657483490, "job": 78, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4497, 4500, 4505, 4509], "score": 3.91865, "input_data_size": 133300180, "oldest_snapshot_seqno": -1} 2024/11/19-15:18:12.032869 139851179493120 [compaction_job.cc:2437] [default] [JOB 78] Compacting 4@1 files to L2, score 3.92, subcompactions 1 : 1 2024/11/19-15:18:12.032876 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 76 Base level 1, inputs: [4497(31MB) 4500(31MB) 4505(31MB) 4509(31MB)] 2024/11/19-15:18:12.033116 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004597.sst) /storage/mytopling/..//tmp/Topling-82038-KM64RY 2024/11/19-15:18:12.179568 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004597.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:18:12.203604 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-01 2024/11/19-15:18:12.203686 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00083/att-01/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:18:12.214166 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004597.sst): index pass time = 0.03's, 214.495'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463349 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:18:12.279931 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004597.sst): first pass time = 0.25's, 271.925'MB/sec 2024/11/19-15:18:12.279985 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004597.sst): old prealloc_size = 73819750, real_size = 55611248 2024/11/19-15:18:12.280209 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004597.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463349 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.83 avg-zval = 117.87 2024/11/19-15:18:12.280264 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004597.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.101 sec, Key+Value = 0.067 GB 2024/11/19-15:18:12.286667 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-02 2024/11/19-15:18:12.286749 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00079/att-02/compact.done) = No such file or directory: 11.230 sec on http probe failed: 5, giveup 2024/11/19-15:18:12.286766 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00079/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.541 (submit 306.176 ms wait 11.230 s), input 127.820 MiB, send shutdown 2024/11/19-15:18:12.287470 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-00 2024/11/19-15:18:12.287553 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00085/att-00/compact.done) = No such file or directory: 11.205 sec on http probe failed: 5, giveup 2024/11/19-15:18:12.287573 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00085/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.905 sec, real 11.504 (submit 290.297 ms wait 11.205 s), input 126.905 MiB, send shutdown 2024/11/19-15:18:12.294164 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00079/att-02 2024/11/19-15:18:12.294398 139850978166528 [dcompact_etcd.cc:1773] job-00079/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:12.296292 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00079) = success 2024/11/19-15:18:12.296300 139850978166528 [dcompact_etcd.cc:1773] job-00079/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:18:12.296322 139850978166528 (Original Log Time 2024/11/19-15:17:37.681521) EVENT_LOG_v1 {"time_micros": 1732000657681515, "job": 79, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4510, 4511, 4512, 4513], "score": 3.669, "input_data_size": 133300180, "oldest_snapshot_seqno": 116989711} 2024/11/19-15:18:12.296320 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-00 2024/11/19-15:18:12.296329 139850978166528 [compaction_job.cc:2437] [default] [JOB 79] Compacting 4@1 files to L2, score 3.67, subcompactions 1 : 1 2024/11/19-15:18:12.296338 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 77 Base level 1, inputs: [4510(31MB) 4511(31MB) 4512(31MB) 4513(31MB)] 2024/11/19-15:18:12.296535 139851148035840 [dcompact_etcd.cc:1773] job-00085/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:12.296574 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004598.sst) /storage/mytopling/..//tmp/Topling-82038-kwmLA0 2024/11/19-15:18:12.374314 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004597.sst): second pass time = 0.00's,118717600.877'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 588.480'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 4004.539'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.597'MB/sec write SST data time = 0.05's, 1121.646'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 118717600.877'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463349 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.83 avg-zval = 117.87 usrkeys = 463349 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.83 avg-zval = 117.87 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6616 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5032 value = 0.9913 dict = 0.00 all = 1.2092 } Zip/UnZip{ index = 0.0952 value = 1.0088 dict = 0.00 all = 0.8270 } ---------------------------- total value len = 3.167611 GB avg = 0.303 KB (by entry num) total key len = 0.265626 GB avg = 0.025 KB total ukey len = 0.182102 GB avg = 0.017 KB total ukey num = 0.010440479 Billion total entry num = 0.010440479 Billion write speed all = 17.370843455 MB/sec (with seq num) write speed all = 16.948245381 MB/sec (without seq num) 2024/11/19-15:18:12.377712 139851179493120 [compaction_job.cc:1948] [default] [JOB 78] Generated table #4597: 463349 keys, 55500696 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:12.377763 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000692377740, "cf_name": "default", "job": 78, "event": "table_file_creation", "file_number": 4597, "file_size": 55500696, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54612896, "index_size": 882304, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12973772, "raw_average_key_size": 28, "raw_value_size": 54135226, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463349, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000692, "oldest_key_time": 0, "file_creation_time": 1732000692, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4597, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:12.391671 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004599.sst) /storage/mytopling/..//tmp/Topling-82038-5b1dSW 2024/11/19-15:18:12.444315 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004598.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:18:12.499627 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004598.sst): index pass time = 0.06's, 134.120'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463333 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:18:12.537854 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004599.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:18:12.560516 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004598.sst): first pass time = 0.26's, 254.271'MB/sec 2024/11/19-15:18:12.560570 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004598.sst): old prealloc_size = 73819750, real_size = 55604128 2024/11/19-15:18:12.560800 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004598.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463333 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.84 avg-zval = 117.87 2024/11/19-15:18:12.560853 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004598.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0011 GB, waited 0.117 sec, Key+Value = 0.067 GB 2024/11/19-15:18:12.572015 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004599.sst): index pass time = 0.03's, 216.731'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462275 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:18:12.607283 139851148035840 [dcompact_etcd.cc:1163] job-00085/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.311 sec, input = 126.905 MiB 2024/11/19-15:18:12.640056 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004599.sst): first pass time = 0.25's, 269.433'MB/sec 2024/11/19-15:18:12.640105 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004599.sst): old prealloc_size = 73819750, real_size = 55473680 2024/11/19-15:18:12.640334 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004599.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462275 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.76 avg-zval = 117.85 2024/11/19-15:18:12.640388 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004599.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.103 sec, Key+Value = 0.067 GB 2024/11/19-15:18:12.653561 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004598.sst): second pass time = 0.00's,115922179.872'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 596.836'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3907.188'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.646'MB/sec write SST data time = 0.05's, 1132.984'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 115922179.872'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463333 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.84 avg-zval = 117.87 usrkeys = 463333 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.84 avg-zval = 117.87 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6616 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5726 value = 0.9913 dict = 0.00 all = 1.2093 } Zip/UnZip{ index = 0.0946 value = 1.0088 dict = 0.00 all = 0.8269 } ---------------------------- total value len = 3.221747 GB avg = 0.295 KB (by entry num) total key len = 0.278599 GB avg = 0.026 KB total ukey len = 0.191369 GB avg = 0.018 KB total ukey num = 0.010903812 Billion total entry num = 0.010903812 Billion write speed all = 17.685400852 MB/sec (with seq num) write speed all = 17.244671213 MB/sec (without seq num) 2024/11/19-15:18:12.656958 139850978166528 [compaction_job.cc:1948] [default] [JOB 79] Generated table #4598: 463333 keys, 55493592 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:12.657009 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000692656986, "cf_name": "default", "job": 79, "event": "table_file_creation", "file_number": 4598, "file_size": 55493592, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54611600, "index_size": 876480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12973324, "raw_average_key_size": 28, "raw_value_size": 54135658, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463333, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000692, "oldest_key_time": 0, "file_creation_time": 1732000692, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4598, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:12.670567 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004600.sst) /storage/mytopling/..//tmp/Topling-82038-HVFIQx 2024/11/19-15:18:12.735524 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004599.sst): second pass time = 0.00's,170268119.874'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 580.312'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3895.273'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.606'MB/sec write SST data time = 0.05's, 1089.297'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 170268119.874'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462275 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 117.85 usrkeys = 462275 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6595 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0554 GB } UnZip/Zip{ index = 10.5024 value = 0.9908 dict = 0.00 all = 1.2087 } Zip/UnZip{ index = 0.0952 value = 1.0093 dict = 0.00 all = 0.8273 } ---------------------------- total value len = 3.275722 GB avg = 0.288 KB (by entry num) total key len = 0.291543 GB avg = 0.026 KB total ukey len = 0.200614 GB avg = 0.018 KB total ukey num = 0.011366087 Billion total entry num = 0.011366087 Billion write speed all = 18.016045338 MB/sec (with seq num) write speed all = 17.556820810 MB/sec (without seq num) 2024/11/19-15:18:12.738849 139851179493120 [compaction_job.cc:1948] [default] [JOB 78] Generated table #4599: 462275 keys, 55363416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:12.738901 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000692738878, "cf_name": "default", "job": 78, "event": "table_file_creation", "file_number": 4599, "file_size": 55363416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54477632, "index_size": 880320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12943700, "raw_average_key_size": 28, "raw_value_size": 53974994, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462275, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000692, "oldest_key_time": 0, "file_creation_time": 1732000692, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4599, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:12.752145 139851179493120 [compaction_job.cc:703] job-00078: subcompact[0], size: 0.110864 G, files: /storage/mytopling/.rocksdb [004597,004599] 2024/11/19-15:18:12.753314 139851179493120 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:18:12.753450 139851179493120 (Original Log Time 2024/11/19-15:18:12.032909) EVENT_LOG_v1 {"time_micros": 1732000692032889, "job": 78, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4497, 4500, 4505, 4509], "score": 3.91865, "input_data_size": 133300180, "oldest_snapshot_seqno": -1} 2024/11/19-15:18:12.753457 139851179493120 (Original Log Time 2024/11/19-15:18:12.753099) [compaction_job.cc:2022] [default] [JOB 78] Compacted 4@1 files to L2 => 110864112 bytes 2024/11/19-15:18:12.753461 139851179493120 (Original Log Time 2024/11/19-15:18:12.753282) [compaction_job.cc:1224] [default] compacted to: files[4 61 20 0 0 0 0] max score 2.27 (41 files need compaction), MB/sec: 185.3 rd, 154.1 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925624, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:12.753478 139851179493120 (Original Log Time 2024/11/19-15:18:12.753308) EVENT_LOG_v1 {"time_micros": 1732000692753296, "cf": "default", "job": 78, "event": "compaction_finished", "compaction_time_micros": 719250, "compaction_time_cpu_micros": 713131, "output_level": 2, "num_output_files": 2, "total_output_size": 110864112, "num_input_records": 925624, "num_output_records": 925624, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 61, 20, 0, 0, 0, 0]} 2024/11/19-15:18:12.757536 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004472.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:12.757564 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000692757561, "job": 78, "event": "table_file_deletion", "file_number": 4472} 2024/11/19-15:18:12.758141 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004481.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:12.758154 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000692758153, "job": 78, "event": "table_file_deletion", "file_number": 4481} 2024/11/19-15:18:12.760573 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004483.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:12.760594 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000692760592, "job": 78, "event": "table_file_deletion", "file_number": 4483} 2024/11/19-15:18:12.761270 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004491.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:12.761284 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000692761283, "job": 78, "event": "table_file_deletion", "file_number": 4491} 2024/11/19-15:18:12.761394 139851179493120 [compaction_job.cc:2437] [default] [JOB 94] Compacting 4@1 files to L2, score 2.27, subcompactions 1 : 1 2024/11/19-15:18:12.761411 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 92 Base level 1, inputs: [4536(31MB) 4537(31MB) 4538(31MB) 4539(31MB)] 2024/11/19-15:18:12.779995 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-02 2024/11/19-15:18:12.780082 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00081/att-02/compact.done) = No such file or directory: 6.576 sec on http probe failed: 4, retry 2024/11/19-15:18:12.816008 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004600.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:18:12.862657 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004600.sst): index pass time = 0.05's, 158.652'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462271 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:18:12.883091 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-01 2024/11/19-15:18:12.883177 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00084/att-01/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:18:12.928912 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004600.sst): first pass time = 0.26's, 259.054'MB/sec 2024/11/19-15:18:12.928983 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004600.sst): old prealloc_size = 73819750, real_size = 55473856 2024/11/19-15:18:12.929215 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004600.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462271 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.76 avg-zval = 117.85 2024/11/19-15:18:12.929271 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004600.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.113 sec, Key+Value = 0.067 GB 2024/11/19-15:18:13.022802 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004600.sst): second pass time = 0.00's,103801642.308'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 590.197'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3888.536'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.602'MB/sec write SST data time = 0.05's, 1125.974'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 103801642.308'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462271 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 117.85 usrkeys = 462271 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6596 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0554 GB } UnZip/Zip{ index = 10.5016 value = 0.9908 dict = 0.00 all = 1.2087 } Zip/UnZip{ index = 0.0952 value = 1.0093 dict = 0.00 all = 0.8273 } ---------------------------- total value len = 3.329699 GB avg = 0.282 KB (by entry num) total key len = 0.304486 GB avg = 0.026 KB total ukey len = 0.209860 GB avg = 0.018 KB total ukey num = 0.011828358 Billion total entry num = 0.011828358 Billion write speed all = 18.327428584 MB/sec (with seq num) write speed all = 17.850219261 MB/sec (without seq num) 2024/11/19-15:18:13.026208 139850978166528 [compaction_job.cc:1948] [default] [JOB 79] Generated table #4600: 462271 keys, 55363608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:13.026262 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000693026237, "cf_name": "default", "job": 79, "event": "table_file_creation", "file_number": 4600, "file_size": 55363608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54477744, "index_size": 880384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12943588, "raw_average_key_size": 28, "raw_value_size": 53976854, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462271, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000692, "oldest_key_time": 0, "file_creation_time": 1732000692, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4600, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:13.039958 139850978166528 [compaction_job.cc:703] job-00079: subcompact[0], size: 0.110857 G, files: /storage/mytopling/.rocksdb [004598,004600] 2024/11/19-15:18:13.041068 139850978166528 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:18:13.041172 139850978166528 (Original Log Time 2024/11/19-15:18:12.296371) EVENT_LOG_v1 {"time_micros": 1732000692296351, "job": 79, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4510, 4511, 4512, 4513], "score": 3.669, "input_data_size": 133300180, "oldest_snapshot_seqno": 116989711} 2024/11/19-15:18:13.041178 139850978166528 (Original Log Time 2024/11/19-15:18:13.040830) [compaction_job.cc:2022] [default] [JOB 79] Compacted 4@1 files to L2 => 110857200 bytes 2024/11/19-15:18:13.041182 139850978166528 (Original Log Time 2024/11/19-15:18:13.041034) [compaction_job.cc:1224] [default] compacted to: files[4 57 22 0 0 0 0] max score 2.02 (37 files need compaction), MB/sec: 179.3 rd, 149.1 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925604, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:13.041198 139850978166528 (Original Log Time 2024/11/19-15:18:13.041061) EVENT_LOG_v1 {"time_micros": 1732000693041047, "cf": "default", "job": 79, "event": "compaction_finished", "compaction_time_micros": 743602, "compaction_time_cpu_micros": 740645, "output_level": 2, "num_output_files": 2, "total_output_size": 110857200, "num_input_records": 925604, "num_output_records": 925604, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 57, 22, 0, 0, 0, 0]} 2024/11/19-15:18:13.045345 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004486.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:13.045370 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000693045367, "job": 79, "event": "table_file_deletion", "file_number": 4486} 2024/11/19-15:18:13.049350 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004487.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:13.049371 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000693049369, "job": 79, "event": "table_file_deletion", "file_number": 4487} 2024/11/19-15:18:13.053498 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004488.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:13.053520 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000693053518, "job": 79, "event": "table_file_deletion", "file_number": 4488} 2024/11/19-15:18:13.056328 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004490.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:13.056349 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000693056346, "job": 79, "event": "table_file_deletion", "file_number": 4490} 2024/11/19-15:18:13.056474 139850978166528 [compaction_job.cc:2437] [default] [JOB 95] Compacting 4@1 files to L2, score 2.02, subcompactions 1 : 1 2024/11/19-15:18:13.056484 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 93 Base level 1, inputs: [4540(31MB) 4541(31MB) 4542(31MB) 4543(31MB)] 2024/11/19-15:18:13.063248 139851179493120 [dcompact_etcd.cc:1163] job-00094/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.301 sec, input = 126.812 MiB 2024/11/19-15:18:13.351529 139850978166528 [dcompact_etcd.cc:1163] job-00095/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.295 sec, input = 126.833 MiB 2024/11/19-15:18:13.353033 139852362282752 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:18:13.449712 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-01 2024/11/19-15:18:13.449799 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00082/att-01/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:18:14.211639 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-01 2024/11/19-15:18:14.211733 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00083/att-01/compact.done) = No such file or directory: 7.185 sec on http probe failed: 3, retry 2024/11/19-15:18:14.329894 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-02 2024/11/19-15:18:14.329975 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00081/att-02/compact.done) = No such file or directory: 8.125 sec on http probe failed: 5, giveup 2024/11/19-15:18:14.329993 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00081/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 77.023 sec, real 8.461 (submit 325.835 ms wait 8.125 s), input 77.023 MiB, send shutdown 2024/11/19-15:18:14.338280 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00081/att-02 2024/11/19-15:18:14.338508 139851169007360 [dcompact_etcd.cc:1773] job-00081/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:14.341604 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00081) = success 2024/11/19-15:18:14.341613 139851169007360 [dcompact_etcd.cc:1773] job-00081/att-03: CleanFiles 0.003 sec : 0.000 2024/11/19-15:18:14.341634 139851169007360 (Original Log Time 2024/11/19-15:17:48.995915) EVENT_LOG_v1 {"time_micros": 1732000668995908, "job": 81, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4514, 4515, 4516, 4517], "score": 3.41935, "input_data_size": 81911437, "oldest_snapshot_seqno": 117927609} 2024/11/19-15:18:14.341641 139851169007360 [compaction_job.cc:2437] [default] [JOB 81] Compacting 4@1 files to L2, score 3.42, subcompactions 1 : 1 2024/11/19-15:18:14.341649 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 79 Base level 1, inputs: [4514(31MB) 4515(5657KB) 4516(9050KB) 4517(31MB)] 2024/11/19-15:18:14.341891 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004601.sst) /storage/mytopling/..//tmp/Topling-82038-vVclfn 2024/11/19-15:18:14.427163 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004601.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0024 GB, waited 0.000 sec, Key+Value = 0.039 GB 2024/11/19-15:18:14.458647 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004601.sst): index pass time = 0.03's, 137.793'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 270838 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 7 raw-key = 0.0043 GB zip-key = 0.0005 GB avg-key = 16.00 avg-zkey = 1.92 2024/11/19-15:18:14.494255 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004601.sst): first pass time = 0.15's, 257.416'MB/sec 2024/11/19-15:18:14.494310 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004601.sst): old prealloc_size = 73819750, real_size = 32507952 2024/11/19-15:18:14.494469 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004601.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 270838 prefix = 4 raw-key = 0.0043 GB zip-key = 0.0005 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0316 GB zip-val = 0.0319 GB avg-val = 116.80 avg-zval = 117.86 2024/11/19-15:18:14.494534 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004601.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0006 GB, waited 0.067 sec, Key+Value = 0.039 GB 2024/11/19-15:18:14.548674 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004601.sst): second pass time = 0.00's,92767741.935'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 596.957'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3356.892'MB/sec (index lex order gen) rebuild zvType time = 0.03's, 2.650'MB/sec write SST data time = 0.03's, 1132.813'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.032'GB zip my value throughput = 92767741.935'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 270838 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.80 avg-zval = 117.86 usrkeys = 270838 avg-key = 20.00 avg-zkey = 1.92 avg-val = 116.80 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 3869 UnZipSize{ index = 0.0054 GB value = 0.0316 GB dict = 0.00 MB all = 0.0392 GB } __ZipSize{ index = 0.0005 GB value = 0.0319 GB dict = 0.00 MB all = 0.0324 GB } UnZip/Zip{ index = 10.4207 value = 0.9910 dict = 0.00 all = 1.2088 } Zip/UnZip{ index = 0.0960 value = 1.0091 dict = 0.00 all = 0.8273 } ---------------------------- total value len = 3.361333 GB avg = 0.278 KB (by entry num) total key len = 0.312070 GB avg = 0.026 KB total ukey len = 0.215276 GB avg = 0.018 KB total ukey num = 0.012099196 Billion total entry num = 0.012099196 Billion write speed all = 18.383739727 MB/sec (with seq num) write speed all = 17.899331132 MB/sec (without seq num) 2024/11/19-15:18:14.550936 139851169007360 [compaction_job.cc:1948] [default] [JOB 81] Generated table #4601: 270838 keys, 32442712 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:14.550989 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000694550965, "cf_name": "default", "job": 81, "event": "table_file_creation", "file_number": 4601, "file_size": 32442712, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 31920224, "index_size": 519808, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 7583464, "raw_average_key_size": 28, "raw_value_size": 31633800, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 270838, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000694, "oldest_key_time": 0, "file_creation_time": 1732000694, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4601, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:14.558825 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004602.sst) /storage/mytopling/..//tmp/Topling-82038-hxRpYd 2024/11/19-15:18:14.587297 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004602.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.008 GB 2024/11/19-15:18:14.587404 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004602.sst): index pass time = 0.00's,5012.154'MB/sec index type = UintIndex_AllOne usrkeys = 100000 min-keylen = 4 max-keylen = 4 prefix = 4 subcplen = 1 raw-key = 0.0004 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 0.00 2024/11/19-15:18:14.597042 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004602.sst): first pass time = 0.04's, 217.294'MB/sec 2024/11/19-15:18:14.597083 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004602.sst): old prealloc_size = 73819750, real_size = 7014304 2024/11/19-15:18:14.597103 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004602.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 100000 prefix = 4 raw-key = 0.0004 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 0.00 raw-val = 0.0067 GB zip-val = 0.0070 GB avg-val = 67.01 avg-zval = 69.89 2024/11/19-15:18:14.607268 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004602.sst): second pass time = 0.00's,32689043.902'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.01's, 687.353'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.503'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 6.683'MB/sec write SST data time = 0.01's, 1090.199'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.007'GB zip my value throughput = 32689043.902'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 100000 avg-key = 16.00 avg-zkey = 0.00 avg-val = 67.01 avg-zval = 69.89 usrkeys = 100000 avg-key = 8.00 avg-zkey = 0.00 avg-val = 67.01 avg-zval = 69.89 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0067 GB dict = 0.00 MB all = 0.0083 GB } __ZipSize{ index = 0.0000 GB value = 0.0070 GB dict = 0.00 MB all = 0.0070 GB } UnZip/Zip{ index =4166.6667 value = 0.9588 dict = 0.00 all = 1.1875 } Zip/UnZip{ index = 0.0002 value = 1.0429 dict = 0.00 all = 0.8421 } ---------------------------- total value len = 3.368034 GB avg = 0.276 KB (by entry num) total key len = 0.313670 GB avg = 0.026 KB total ukey len = 0.216076 GB avg = 0.018 KB total ukey num = 0.012199196 Billion total entry num = 0.012199196 Billion write speed all = 18.419881609 MB/sec (with seq num) write speed all = 17.931612571 MB/sec (without seq num) 2024/11/19-15:18:14.607977 139851169007360 [compaction_job.cc:1948] [default] [JOB 81] Generated table #4602: 100000 keys, 6990274 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:14.608025 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000694608002, "cf_name": "default", "job": 81, "event": "table_file_creation", "file_number": 4602, "file_size": 6990274, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 6988976, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1600000, "raw_average_key_size": 16, "raw_value_size": 6701254, "raw_average_value_size": 67, "num_data_blocks": 1, "num_entries": 100000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000694, "oldest_key_time": 0, "file_creation_time": 1732000694, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4602, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:14.609828 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004603.sst) /storage/mytopling/..//tmp/Topling-82038-qO6jGG 2024/11/19-15:18:14.636273 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004603.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.033 GB 2024/11/19-15:18:14.636412 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004603.sst): index pass time = 0.00's,2341.342'MB/sec index type = UintIndex_AllOne usrkeys = 32060 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 6 raw-key = 0.0003 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.01 2024/11/19-15:18:14.668971 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004603.sst): first pass time = 0.06's, 562.263'MB/sec 2024/11/19-15:18:14.669019 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004603.sst): old prealloc_size = 73819750, real_size = 32714040 2024/11/19-15:18:14.669043 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004603.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 32060 prefix = 4 raw-key = 0.0003 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.01 raw-val = 0.0326 GB zip-val = 0.0327 GB avg-val =1017.01 avg-zval =1020.14 2024/11/19-15:18:14.695657 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004603.sst): second pass time = 0.00's,114404417.544'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.03's, 1228.580'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.050'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 0.424'MB/sec write SST data time = 0.01's, 4253.952'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.033'GB zip my value throughput = 114404417.544'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 32060 avg-key = 20.00 avg-zkey = 0.01 avg-val = 1017.01 avg-zval = 1020.14 usrkeys = 32060 avg-key = 12.00 avg-zkey = 0.01 avg-val = 1017.01 avg-zval = 1020.14 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0004 GB value = 0.0326 GB dict = 0.00 MB all = 0.0332 GB } __ZipSize{ index = 0.0000 GB value = 0.0327 GB dict = 0.00 MB all = 0.0327 GB } UnZip/Zip{ index =2003.7500 value = 0.9969 dict = 0.00 all = 1.0165 } Zip/UnZip{ index = 0.0005 value = 1.0031 dict = 0.00 all = 0.9838 } ---------------------------- total value len = 3.400639 GB avg = 0.278 KB (by entry num) total key len = 0.314311 GB avg = 0.026 KB total ukey len = 0.216461 GB avg = 0.018 KB total ukey num = 0.012231256 Billion total entry num = 0.012231256 Billion write speed all = 18.578001404 MB/sec (with seq num) write speed all = 18.088665556 MB/sec (without seq num) 2024/11/19-15:18:14.697574 139851169007360 [compaction_job.cc:1948] [default] [JOB 81] Generated table #4603: 32060 keys, 32706954 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:14.697623 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000694697600, "cf_name": "default", "job": 81, "event": "table_file_creation", "file_number": 4603, "file_size": 32706954, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32705664, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32605259, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000694, "oldest_key_time": 0, "file_creation_time": 1732000694, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4603, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:14.704734 139851169007360 [compaction_job.cc:703] job-00081: subcompact[0], size: 0.072140 G, files: /storage/mytopling/.rocksdb [004601,004602,004603] 2024/11/19-15:18:14.705566 139851169007360 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:18:14.705668 139851169007360 (Original Log Time 2024/11/19-15:18:14.341682) EVENT_LOG_v1 {"time_micros": 1732000694341663, "job": 81, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4514, 4515, 4516, 4517], "score": 3.41935, "input_data_size": 81911437, "oldest_snapshot_seqno": 117927609} 2024/11/19-15:18:14.705674 139851169007360 (Original Log Time 2024/11/19-15:18:14.705353) [compaction_job.cc:2022] [default] [JOB 81] Compacted 4@1 files to L2 => 72139940 bytes 2024/11/19-15:18:14.705677 139851169007360 (Original Log Time 2024/11/19-15:18:14.705534) [compaction_job.cc:1224] [default] compacted to: files[4 53 25 0 0 0 0] max score 1.77 (33 files need compaction), MB/sec: 225.6 rd, 198.7 wr, level 2, files in(4, 0) out(3 +0 blob) MB in(78.1, 0.0 +0.0 blob) out(68.8 +0.0 blob), read-write-amplify(1.9) write-amplify(0.9) OK, records in: 402898, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:14.705692 139851169007360 (Original Log Time 2024/11/19-15:18:14.705560) EVENT_LOG_v1 {"time_micros": 1732000694705547, "cf": "default", "job": 81, "event": "compaction_finished", "compaction_time_micros": 363063, "compaction_time_cpu_micros": 359787, "output_level": 2, "num_output_files": 3, "total_output_size": 72139940, "num_input_records": 402898, "num_output_records": 402898, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 53, 25, 0, 0, 0, 0]} 2024/11/19-15:18:14.709839 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004474.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:14.709864 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000694709861, "job": 81, "event": "table_file_deletion", "file_number": 4474} 2024/11/19-15:18:14.713635 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004479.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:14.713656 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000694713654, "job": 81, "event": "table_file_deletion", "file_number": 4479} 2024/11/19-15:18:14.717701 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004503.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:14.717721 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000694717719, "job": 81, "event": "table_file_deletion", "file_number": 4503} 2024/11/19-15:18:14.720054 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004506.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:14.720072 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000694720070, "job": 81, "event": "table_file_deletion", "file_number": 4506} 2024/11/19-15:18:14.720181 139851169007360 [compaction_job.cc:2437] [default] [JOB 96] Compacting 4@1 files to L2, score 1.77, subcompactions 1 : 1 2024/11/19-15:18:14.720189 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 94 Base level 1, inputs: [4544(31MB) 4545(31MB) 4546(31MB) 4547(31MB)] 2024/11/19-15:18:14.887847 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-01 2024/11/19-15:18:14.887927 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00084/att-01/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:18:15.018235 139851169007360 [dcompact_etcd.cc:1163] job-00096/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.298 sec, input = 126.961 MiB 2024/11/19-15:18:15.459556 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-01 2024/11/19-15:18:15.459656 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00082/att-01/compact.done) = No such file or directory: 9.194 sec on http probe failed: 4, retry 2024/11/19-15:18:15.785325 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-01 2024/11/19-15:18:15.785430 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00085/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:18:16.220610 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-01 2024/11/19-15:18:16.220701 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00083/att-01/compact.done) = No such file or directory: 9.193 sec on http probe failed: 4, retry 2024/11/19-15:18:16.244987 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-00 2024/11/19-15:18:16.245079 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00094/att-00/compact.done) = No such file or directory: 3.170 sec on http probe failed: 1, retry 2024/11/19-15:18:16.527777 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-00 2024/11/19-15:18:16.527880 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00095/att-00/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:18:16.896857 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-01 2024/11/19-15:18:16.896938 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00084/att-01/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:18:17.467702 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-01 2024/11/19-15:18:17.467791 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00082/att-01/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:18:17.467811 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00082/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.957 sec, real 11.531 (submit 318.819 ms wait 11.204 s), input 126.957 MiB, send shutdown 2024/11/19-15:18:17.477031 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-01 2024/11/19-15:18:17.477255 139851137550080 [dcompact_etcd.cc:1773] job-00082/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:17.744650 139851137550080 [dcompact_etcd.cc:1163] job-00082/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.267 sec, input = 126.957 MiB 2024/11/19-15:18:17.793465 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-01 2024/11/19-15:18:17.793527 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00085/att-01/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:18:18.201300 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-00 2024/11/19-15:18:18.201385 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00096/att-00/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:18:18.229512 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-01 2024/11/19-15:18:18.229567 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00083/att-01/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:18:18.229585 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00083/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.908 sec, real 11.513 (submit 302.683 ms wait 11.202 s), input 126.908 MiB, send shutdown 2024/11/19-15:18:18.237561 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-01 2024/11/19-15:18:18.237778 139851158521600 [dcompact_etcd.cc:1773] job-00083/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:18.253017 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-00 2024/11/19-15:18:18.253068 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00094/att-00/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:18:18.512438 139851158521600 [dcompact_etcd.cc:1163] job-00083/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.275 sec, input = 126.908 MiB 2024/11/19-15:18:18.536996 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-00 2024/11/19-15:18:18.537072 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00095/att-00/compact.done) = No such file or directory: 5.176 sec on http probe failed: 2, retry 2024/11/19-15:18:18.901862 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-01 2024/11/19-15:18:18.901969 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00084/att-01/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:18:19.799384 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-01 2024/11/19-15:18:19.799496 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00085/att-01/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:18:20.207304 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-00 2024/11/19-15:18:20.207399 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00096/att-00/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:18:20.262002 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-00 2024/11/19-15:18:20.262082 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00094/att-00/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:18:20.546003 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-00 2024/11/19-15:18:20.546094 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00095/att-00/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:18:20.907568 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-01 2024/11/19-15:18:20.907637 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00084/att-01/compact.done) = No such file or directory: 11.200 sec on http probe failed: 5, giveup 2024/11/19-15:18:20.907654 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00084/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.878 sec, real 11.489 (submit 283.338 ms wait 11.200 s), input 126.878 MiB, send shutdown 2024/11/19-15:18:20.912463 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-01 2024/11/19-15:18:20.912690 139851127064320 [dcompact_etcd.cc:1773] job-00084/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:20.927856 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-02 2024/11/19-15:18:20.927918 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00082/att-02/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:18:21.203838 139851127064320 [dcompact_etcd.cc:1163] job-00084/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.291 sec, input = 126.878 MiB 2024/11/19-15:18:21.693276 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-02 2024/11/19-15:18:21.693378 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00083/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:18:21.805578 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-01 2024/11/19-15:18:21.805678 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00085/att-01/compact.done) = No such file or directory: 9.192 sec on http probe failed: 4, retry 2024/11/19-15:18:22.216624 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-00 2024/11/19-15:18:22.216710 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00096/att-00/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:18:22.266668 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-00 2024/11/19-15:18:22.266734 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00094/att-00/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:18:22.555651 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-00 2024/11/19-15:18:22.555736 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00095/att-00/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:18:22.935244 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-02 2024/11/19-15:18:22.935332 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00082/att-02/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:18:22.971063 139852349699840 [db_impl_write.cc:2181] [default] New memtable created with log file: #4604. Immutable memtables: 0. 2024/11/19-15:18:22.971121 139852349699840 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:18:22.971382 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 97] Syncing log #4595 2024/11/19-15:18:22.971471 139850957195008 (Original Log Time 2024/11/19-15:18:22.971173) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:18:22.971482 139850957195008 [flush_job.cc:879] [default] [JOB 97] Flushing memtable with next log file: 4604 2024/11/19-15:18:22.971508 139850957195008 EVENT_LOG_v1 {"time_micros": 1732000702971500, "job": 97, "event": "flush_started", "num_memtables": 1, "num_entries": 1434614, "num_deletes": 0, "total_data_size": 461698232, "memory_usage": 541851016, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:18:22.971515 139850957195008 [flush_job.cc:910] [default] [JOB 97] Level-0 flush table #4605: started 2024/11/19-15:18:22.972164 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000010.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004605.sst in ConvertToSST, mem_size = 516.749 M, time = 636.696 us 2024/11/19-15:18:23.547680 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004605.sst): fsize = 516.751 M, time(ms): open: 0.046, seek: 0.017, finish: 0.048, meta: 0.015, Flush: 0.013, sync: 575.345, close: 0.010, all: 575.494 2024/11/19-15:18:23.548753 139850957195008 [flush_job.cc:1083] [default] [JOB 97] Flush lasted 577282 microseconds, and 414379 cpu microseconds. 2024/11/19-15:18:23.568289 139850957195008 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:18:23.568377 139850957195008 (Original Log Time 2024/11/19-15:18:23.547722) [flush_job.cc:1029] [default] [JOB 97] Level-0 flush table #4605: 541852442 bytes OK (needs compaction) 2024/11/19-15:18:23.568383 139850957195008 (Original Log Time 2024/11/19-15:18:23.548769) [memtable_list.cc:560] [default] Level-0 commit table #4605 started 2024/11/19-15:18:23.568386 139850957195008 (Original Log Time 2024/11/19-15:18:23.568259) [memtable_list.cc:759] [default] Level-0 commit table #4605: memtable #1 done 2024/11/19-15:18:23.568388 139850957195008 (Original Log Time 2024/11/19-15:18:23.568282) EVENT_LOG_v1 {"time_micros": 1732000703568276, "job": 97, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [5, 53, 25, 0, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:18:23.568391 139850957195008 (Original Log Time 2024/11/19-15:18:23.568346) [db_impl_compaction_flush.cc:310] [default] Level summary: files[5 53 25 0 0 0 0] max score 1.54 (30 files need compaction) 2024/11/19-15:18:23.699462 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-02 2024/11/19-15:18:23.699552 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00083/att-02/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:18:23.814885 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-01 2024/11/19-15:18:23.814991 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00085/att-01/compact.done) = No such file or directory: 11.198 sec on http probe failed: 5, giveup 2024/11/19-15:18:23.815011 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00085/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.905 sec, real 11.518 (submit 310.656 ms wait 11.198 s), input 126.905 MiB, send shutdown 2024/11/19-15:18:23.823872 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-01 2024/11/19-15:18:23.824100 139851148035840 [dcompact_etcd.cc:1773] job-00085/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:24.110794 139851148035840 [dcompact_etcd.cc:1163] job-00085/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.287 sec, input = 126.905 MiB 2024/11/19-15:18:24.226428 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-00 2024/11/19-15:18:24.226526 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00096/att-00/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:18:24.275236 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-00 2024/11/19-15:18:24.275318 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00094/att-00/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:18:24.275337 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00094/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.812 sec, real 11.513 (submit 301.377 ms wait 11.204 s), input 126.812 MiB, send shutdown 2024/11/19-15:18:24.283539 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-00 2024/11/19-15:18:24.283751 139851179493120 [dcompact_etcd.cc:1773] job-00094/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:24.384543 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-02 2024/11/19-15:18:24.384614 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00084/att-02/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:18:24.547276 139851179493120 [dcompact_etcd.cc:1163] job-00094/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.263 sec, input = 126.812 MiB 2024/11/19-15:18:24.562828 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-00 2024/11/19-15:18:24.562887 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00095/att-00/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:18:24.562903 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00095/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.833 sec, real 11.506 (submit 294.561 ms wait 11.204 s), input 126.833 MiB, send shutdown 2024/11/19-15:18:24.571072 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-00 2024/11/19-15:18:24.571297 139850978166528 [dcompact_etcd.cc:1773] job-00095/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:24.851548 139850978166528 [dcompact_etcd.cc:1163] job-00095/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 126.833 MiB 2024/11/19-15:18:24.943441 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-02 2024/11/19-15:18:24.943503 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00082/att-02/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:18:25.709011 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-02 2024/11/19-15:18:25.709097 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00083/att-02/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:18:26.235425 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-00 2024/11/19-15:18:26.235518 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00096/att-00/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:18:26.235537 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00096/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.961 sec, real 11.515 (submit 297.596 ms wait 11.208 s), input 126.961 MiB, send shutdown 2024/11/19-15:18:26.243959 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-00 2024/11/19-15:18:26.244176 139851169007360 [dcompact_etcd.cc:1773] job-00096/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:26.393845 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-02 2024/11/19-15:18:26.393918 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00084/att-02/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:18:26.528011 139851169007360 [dcompact_etcd.cc:1163] job-00096/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.284 sec, input = 126.961 MiB 2024/11/19-15:18:26.952181 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-02 2024/11/19-15:18:26.952270 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00082/att-02/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:18:27.288727 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-02 2024/11/19-15:18:27.288799 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00085/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:18:27.714167 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-02 2024/11/19-15:18:27.714253 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00083/att-02/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:18:27.726336 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-01 2024/11/19-15:18:27.726410 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00094/att-01/compact.done) = No such file or directory: 3.170 sec on http probe failed: 1, retry 2024/11/19-15:18:28.032899 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-01 2024/11/19-15:18:28.032990 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00095/att-01/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:18:28.398831 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-02 2024/11/19-15:18:28.398905 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00084/att-02/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:18:28.961769 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-02 2024/11/19-15:18:28.961842 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00082/att-02/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:18:28.961859 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00082/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.957 sec, real 11.485 (submit 267.315 ms wait 11.208 s), input 126.957 MiB, send shutdown 2024/11/19-15:18:28.970094 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00082/att-02 2024/11/19-15:18:28.970312 139851137550080 [dcompact_etcd.cc:1773] job-00082/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:28.972198 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00082) = success 2024/11/19-15:18:28.972208 139851137550080 [dcompact_etcd.cc:1773] job-00082/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:18:28.972228 139851137550080 (Original Log Time 2024/11/19-15:17:54.435188) EVENT_LOG_v1 {"time_micros": 1732000674435181, "job": 82, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4518, 4519, 4520, 4521], "score": 3.26644, "input_data_size": 134099656, "oldest_snapshot_seqno": 118433665} 2024/11/19-15:18:28.972235 139851137550080 [compaction_job.cc:2437] [default] [JOB 82] Compacting 4@1 files to L2, score 3.27, subcompactions 1 : 1 2024/11/19-15:18:28.972243 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 80 Base level 1, inputs: [4518(31MB) 4519(31MB) 4520(31MB) 4521(31MB)] 2024/11/19-15:18:28.972489 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004606.sst) /storage/mytopling/..//tmp/Topling-82038-7oqr61 2024/11/19-15:18:29.026210 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004606.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:18:29.026354 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004606.sst): index pass time = 0.00's,4811.867'MB/sec index type = UintIndex_AllOne usrkeys = 64713 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:18:29.092204 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004606.sst): first pass time = 0.12's, 560.642'MB/sec 2024/11/19-15:18:29.092258 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004606.sst): old prealloc_size = 73819750, real_size = 66042456 2024/11/19-15:18:29.092281 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004606.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64713 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.04 avg-zval =1020.29 2024/11/19-15:18:29.146638 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004606.sst): second pass time = 0.00's,162106943.350'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1214.514'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.727'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.415'MB/sec write SST data time = 0.02's, 4298.497'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 162106943.350'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64713 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.04 avg-zval = 1020.29 usrkeys = 64713 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.04 avg-zval = 1020.29 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.5625 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 3.466455 GB avg = 0.282 KB (by entry num) total key len = 0.315605 GB avg = 0.026 KB total ukey len = 0.217238 GB avg = 0.018 KB total ukey num = 0.012295969 Billion total entry num = 0.012295969 Billion write speed all = 17.638889184 MB/sec (with seq num) write speed all = 17.180118591 MB/sec (without seq num) 2024/11/19-15:18:29.150250 139851137550080 [compaction_job.cc:1948] [default] [JOB 82] Generated table #4606: 64713 keys, 66027274 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:29.150302 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000709150278, "cf_name": "default", "job": 82, "event": "table_file_creation", "file_number": 4606, "file_size": 66027274, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025952, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294260, "raw_average_key_size": 20, "raw_value_size": 65815419, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64713, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000708, "oldest_key_time": 0, "file_creation_time": 1732000708, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4606, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:29.164479 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004607.sst) /storage/mytopling/..//tmp/Topling-82038-b0Uz9d 2024/11/19-15:18:29.216596 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004607.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:18:29.221104 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004607.sst): index pass time = 0.00's, 114.077'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63659 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:18:29.281213 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004607.sst): first pass time = 0.12's, 565.608'MB/sec 2024/11/19-15:18:29.281271 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004607.sst): old prealloc_size = 73819750, real_size = 65072944 2024/11/19-15:18:29.281329 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004607.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63659 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1017.00 avg-zval =1020.26 2024/11/19-15:18:29.281389 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004607.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.065 sec, Key+Value = 0.066 GB 2024/11/19-15:18:29.300629 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-02 2024/11/19-15:18:29.300731 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00085/att-02/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:18:29.341167 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004607.sst): second pass time = 0.00's,275495076.596'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1086.382'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2082.240'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.368'MB/sec write SST data time = 0.02's, 3932.596'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 275495076.596'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63659 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.26 usrkeys = 63659 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7933 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0651 GB } UnZip/Zip{ index = 7.0461 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 3.531196 GB avg = 0.286 KB (by entry num) total key len = 0.316879 GB avg = 0.026 KB total ukey len = 0.218002 GB avg = 0.018 KB total ukey num = 0.012359628 Billion total entry num = 0.012359628 Billion write speed all = 17.930502387 MB/sec (with seq num) write speed all = 17.469774629 MB/sec (without seq num) 2024/11/19-15:18:29.344627 139851137550080 [compaction_job.cc:1948] [default] [JOB 82] Generated table #4607: 63659 keys, 65058436 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:29.344676 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000709344654, "cf_name": "default", "job": 82, "event": "table_file_creation", "file_number": 4607, "file_size": 65058436, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64948448, "index_size": 108416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1273180, "raw_average_key_size": 20, "raw_value_size": 64741343, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63659, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000709, "oldest_key_time": 0, "file_creation_time": 1732000709, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4607, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:29.358577 139851137550080 [compaction_job.cc:703] job-00082: subcompact[0], size: 0.131086 G, files: /storage/mytopling/.rocksdb [004606,004607] 2024/11/19-15:18:29.359188 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:18:29.359344 139851137550080 (Original Log Time 2024/11/19-15:18:28.972276) EVENT_LOG_v1 {"time_micros": 1732000708972256, "job": 82, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4518, 4519, 4520, 4521], "score": 3.26644, "input_data_size": 134099656, "oldest_snapshot_seqno": 118433665} 2024/11/19-15:18:29.359351 139851137550080 (Original Log Time 2024/11/19-15:18:29.358975) [compaction_job.cc:2022] [default] [JOB 82] Compacted 4@1 files to L2 => 131085710 bytes 2024/11/19-15:18:29.359354 139851137550080 (Original Log Time 2024/11/19-15:18:29.359157) [compaction_job.cc:1224] [default] compacted to: files[5 49 27 0 0 0 0] max score 1.67 (30 files need compaction), MB/sec: 347.1 rd, 339.3 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128372, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:29.359372 139851137550080 (Original Log Time 2024/11/19-15:18:29.359182) EVENT_LOG_v1 {"time_micros": 1732000709359170, "cf": "default", "job": 82, "event": "compaction_finished", "compaction_time_micros": 386312, "compaction_time_cpu_micros": 384719, "output_level": 2, "num_output_files": 2, "total_output_size": 131085710, "num_input_records": 128372, "num_output_records": 128372, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 49, 27, 0, 0, 0, 0]} 2024/11/19-15:18:29.363135 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004492.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:29.363160 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000709363157, "job": 82, "event": "table_file_deletion", "file_number": 4492} 2024/11/19-15:18:29.367312 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004494.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:29.367333 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000709367331, "job": 82, "event": "table_file_deletion", "file_number": 4494} 2024/11/19-15:18:29.371779 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004496.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:29.371799 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000709371797, "job": 82, "event": "table_file_deletion", "file_number": 4496} 2024/11/19-15:18:29.375877 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004499.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:29.375896 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000709375894, "job": 82, "event": "table_file_deletion", "file_number": 4499} 2024/11/19-15:18:29.376021 139851137550080 [compaction_job.cc:2437] [default] [JOB 98] Compacting 1@1 files to L2, score 0.78, subcompactions 1 : 1 2024/11/19-15:18:29.376029 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 96 Base level 1, inputs: [4564(31MB)] 2024/11/19-15:18:29.674326 139851137550080 [dcompact_etcd.cc:1163] job-00098/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.298 sec, input = 31.699 MiB 2024/11/19-15:18:29.711460 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-01 2024/11/19-15:18:29.711529 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00096/att-01/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:18:29.721291 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-02 2024/11/19-15:18:29.721348 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00083/att-02/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:18:29.721363 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00083/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.908 sec, real 11.484 (submit 274.586 ms wait 11.202 s), input 126.908 MiB, send shutdown 2024/11/19-15:18:29.729228 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00083/att-02 2024/11/19-15:18:29.729432 139851158521600 [dcompact_etcd.cc:1773] job-00083/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:29.731226 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00083) = success 2024/11/19-15:18:29.731234 139851158521600 [dcompact_etcd.cc:1773] job-00083/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:18:29.731253 139851158521600 (Original Log Time 2024/11/19-15:17:55.206489) EVENT_LOG_v1 {"time_micros": 1732000675206482, "job": 83, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4522, 4523, 4524, 4525], "score": 3.01666, "input_data_size": 134099592, "oldest_snapshot_seqno": -1} 2024/11/19-15:18:29.731260 139851158521600 [compaction_job.cc:2437] [default] [JOB 83] Compacting 4@1 files to L2, score 3.02, subcompactions 1 : 1 2024/11/19-15:18:29.731268 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 81 Base level 1, inputs: [4522(31MB) 4523(31MB) 4524(31MB) 4525(31MB)] 2024/11/19-15:18:29.731511 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004608.sst) /storage/mytopling/..//tmp/Topling-82038-3hdrQY 2024/11/19-15:18:29.735497 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-01 2024/11/19-15:18:29.735547 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00094/att-01/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:18:29.795966 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004608.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:18:29.799471 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004608.sst): index pass time = 0.00's, 149.182'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64716 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:18:29.862240 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004608.sst): first pass time = 0.13's, 513.401'MB/sec 2024/11/19-15:18:29.862290 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004608.sst): old prealloc_size = 73819750, real_size = 66152176 2024/11/19-15:18:29.862345 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004608.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64716 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.98 avg-zval =1020.24 2024/11/19-15:18:29.862399 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004608.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.067 GB 2024/11/19-15:18:29.921593 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004608.sst): second pass time = 0.00's,205672115.625'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1115.571'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2173.685'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.380'MB/sec write SST data time = 0.02's, 3981.042'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 205672115.625'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64716 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 1020.24 usrkeys = 64716 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0466 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 3.597011 GB avg = 0.290 KB (by entry num) total key len = 0.318173 GB avg = 0.026 KB total ukey len = 0.218778 GB avg = 0.018 KB total ukey num = 0.012424344 Billion total entry num = 0.012424344 Billion write speed all = 18.193999128 MB/sec (with seq num) write speed all = 17.732108167 MB/sec (without seq num) 2024/11/19-15:18:29.925106 139851158521600 [compaction_job.cc:1948] [default] [JOB 83] Generated table #4608: 64716 keys, 66137412 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:29.925159 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000709925136, "cf_name": "default", "job": 83, "event": "table_file_creation", "file_number": 4608, "file_size": 66137412, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025632, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294320, "raw_average_key_size": 20, "raw_value_size": 65815077, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64716, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000709, "oldest_key_time": 0, "file_creation_time": 1732000709, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4608, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:29.939330 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004609.sst) /storage/mytopling/..//tmp/Topling-82038-MajqGW 2024/11/19-15:18:29.992544 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004609.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:18:29.992656 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004609.sst): index pass time = 0.00's,6207.238'MB/sec index type = UintIndex_AllOne usrkeys = 63611 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:18:30.042510 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-01 2024/11/19-15:18:30.042603 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00095/att-01/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:18:30.057670 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004609.sst): first pass time = 0.12's, 557.476'MB/sec 2024/11/19-15:18:30.057716 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004609.sst): old prealloc_size = 73819750, real_size = 64914184 2024/11/19-15:18:30.057738 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004609.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63611 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1016.98 avg-zval =1020.23 2024/11/19-15:18:30.111031 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004609.sst): second pass time = 0.00's,139721349.892'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1217.719'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.996'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.415'MB/sec write SST data time = 0.01's, 4330.756'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 139721349.892'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63611 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.98 avg-zval = 1020.23 usrkeys = 63611 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.98 avg-zval = 1020.23 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0000 GB value = 0.0649 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index =3975.6875 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 3.661702 GB avg = 0.293 KB (by entry num) total key len = 0.319445 GB avg = 0.026 KB total ukey len = 0.219542 GB avg = 0.018 KB total ukey num = 0.012487955 Billion total entry num = 0.012487955 Billion write speed all = 18.484260401 MB/sec (with seq num) write speed all = 18.020412954 MB/sec (without seq num) 2024/11/19-15:18:30.114608 139851158521600 [compaction_job.cc:1948] [default] [JOB 83] Generated table #4609: 63611 keys, 64899274 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:30.114659 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000710114636, "cf_name": "default", "job": 83, "event": "table_file_creation", "file_number": 4609, "file_size": 64899274, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64897936, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1272220, "raw_average_key_size": 20, "raw_value_size": 64690985, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63611, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000709, "oldest_key_time": 0, "file_creation_time": 1732000709, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4609, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:30.128783 139851158521600 [compaction_job.cc:703] job-00083: subcompact[0], size: 0.131037 G, files: /storage/mytopling/.rocksdb [004608,004609] 2024/11/19-15:18:30.129404 139851158521600 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:18:30.129561 139851158521600 (Original Log Time 2024/11/19-15:18:29.731295) EVENT_LOG_v1 {"time_micros": 1732000709731280, "job": 83, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4522, 4523, 4524, 4525], "score": 3.01666, "input_data_size": 134099592, "oldest_snapshot_seqno": -1} 2024/11/19-15:18:30.129572 139851158521600 (Original Log Time 2024/11/19-15:18:30.129198) [compaction_job.cc:2022] [default] [JOB 83] Compacted 4@1 files to L2 => 131036686 bytes 2024/11/19-15:18:30.129578 139851158521600 (Original Log Time 2024/11/19-15:18:30.129372) [compaction_job.cc:1224] [default] compacted to: files[5 45 29 0 0 0 0] max score 1.82 (29 files need compaction), MB/sec: 337.4 rd, 329.7 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128327, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:30.129603 139851158521600 (Original Log Time 2024/11/19-15:18:30.129398) EVENT_LOG_v1 {"time_micros": 1732000710129384, "cf": "default", "job": 83, "event": "compaction_finished", "compaction_time_micros": 397500, "compaction_time_cpu_micros": 395534, "output_level": 2, "num_output_files": 2, "total_output_size": 131036686, "num_input_records": 128327, "num_output_records": 128327, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 45, 29, 0, 0, 0, 0]} 2024/11/19-15:18:30.133415 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004485.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:30.133439 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000710133436, "job": 83, "event": "table_file_deletion", "file_number": 4485} 2024/11/19-15:18:30.137199 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004489.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:30.137219 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000710137217, "job": 83, "event": "table_file_deletion", "file_number": 4489} 2024/11/19-15:18:30.141295 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004493.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:30.141314 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000710141312, "job": 83, "event": "table_file_deletion", "file_number": 4493} 2024/11/19-15:18:30.145201 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004495.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:30.145220 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000710145218, "job": 83, "event": "table_file_deletion", "file_number": 4495} 2024/11/19-15:18:30.145329 139851158521600 [compaction_job.cc:2437] [default] [JOB 99] Compacting 1@1 files to L2, score 0.86, subcompactions 1 : 1 2024/11/19-15:18:30.145337 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 97 Base level 1, inputs: [4552(31MB)] 2024/11/19-15:18:30.407565 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-02 2024/11/19-15:18:30.407639 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00084/att-02/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:18:30.458021 139851158521600 [dcompact_etcd.cc:1163] job-00099/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.312 sec, input = 31.704 MiB 2024/11/19-15:18:30.475225 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-00 2024/11/19-15:18:30.475288 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00098/att-00/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:18:31.117521 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-00 2024/11/19-15:18:31.117595 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00098/att-00/compact.done) = No such file or directory: 1.435 sec on http probe failed: 2, retry 2024/11/19-15:18:31.256056 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-00 2024/11/19-15:18:31.256158 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00099/att-00/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:18:31.309549 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-02 2024/11/19-15:18:31.309635 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00085/att-02/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:18:31.717155 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-01 2024/11/19-15:18:31.717235 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00096/att-01/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:18:31.741142 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-01 2024/11/19-15:18:31.741194 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00094/att-01/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:18:31.759952 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-00 2024/11/19-15:18:31.760009 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00098/att-00/compact.done) = No such file or directory: 2.077 sec on http probe failed: 3, retry 2024/11/19-15:18:31.899500 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-00 2024/11/19-15:18:31.899579 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00099/att-00/compact.done) = No such file or directory: 1.432 sec on http probe failed: 2, retry 2024/11/19-15:18:32.051439 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-01 2024/11/19-15:18:32.051529 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00095/att-01/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:18:32.399685 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-00 2024/11/19-15:18:32.399780 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00098/att-00/compact.done) = No such file or directory: 2.720 sec on http probe failed: 4, retry 2024/11/19-15:18:32.415913 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-02 2024/11/19-15:18:32.415998 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00084/att-02/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:18:32.416015 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00084/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.878 sec, real 11.503 (submit 291.074 ms wait 11.204 s), input 126.878 MiB, send shutdown 2024/11/19-15:18:32.424513 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00084/att-02 2024/11/19-15:18:32.424738 139851127064320 [dcompact_etcd.cc:1773] job-00084/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:32.426605 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00084) = success 2024/11/19-15:18:32.426614 139851127064320 [dcompact_etcd.cc:1773] job-00084/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:18:32.426634 139851127064320 (Original Log Time 2024/11/19-15:17:57.934836) EVENT_LOG_v1 {"time_micros": 1732000677934829, "job": 84, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4526, 4527, 4528, 4529], "score": 2.76688, "input_data_size": 134099016, "oldest_snapshot_seqno": -1} 2024/11/19-15:18:32.426642 139851127064320 [compaction_job.cc:2437] [default] [JOB 84] Compacting 4@1 files to L2, score 2.77, subcompactions 1 : 1 2024/11/19-15:18:32.426649 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 82 Base level 1, inputs: [4526(31MB) 4527(31MB) 4528(31MB) 4529(31MB)] 2024/11/19-15:18:32.426890 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004610.sst) /storage/mytopling/..//tmp/Topling-82038-4CMn7n 2024/11/19-15:18:32.479378 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004610.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:18:32.482686 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004610.sst): index pass time = 0.00's, 157.925'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64716 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:18:32.541851 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-00 2024/11/19-15:18:32.541925 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00099/att-00/compact.done) = No such file or directory: 2.076 sec on http probe failed: 3, retry 2024/11/19-15:18:32.545064 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004610.sst): first pass time = 0.12's, 567.946'MB/sec 2024/11/19-15:18:32.545111 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004610.sst): old prealloc_size = 73819750, real_size = 66152464 2024/11/19-15:18:32.545166 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004610.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64716 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.99 avg-zval =1020.24 2024/11/19-15:18:32.545223 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004610.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.067 GB 2024/11/19-15:18:32.604916 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004610.sst): second pass time = 0.00's,137689048.117'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1106.202'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2147.509'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.376'MB/sec write SST data time = 0.02's, 3952.589'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 137689048.117'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64716 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 usrkeys = 64716 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8063 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0466 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 3.727517 GB avg = 0.297 KB (by entry num) total key len = 0.320739 GB avg = 0.026 KB total ukey len = 0.220318 GB avg = 0.018 KB total ukey num = 0.012552671 Billion total entry num = 0.012552671 Billion write speed all = 18.580701503 MB/sec (with seq num) write speed all = 18.119787193 MB/sec (without seq num) 2024/11/19-15:18:32.608430 139851127064320 [compaction_job.cc:1948] [default] [JOB 84] Generated table #4610: 64716 keys, 66137668 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:32.608483 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000712608458, "cf_name": "default", "job": 84, "event": "table_file_creation", "file_number": 4610, "file_size": 66137668, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025920, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294320, "raw_average_key_size": 20, "raw_value_size": 65815365, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64716, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000712, "oldest_key_time": 0, "file_creation_time": 1732000712, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4610, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:32.622684 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004611.sst) /storage/mytopling/..//tmp/Topling-82038-xe1UzF 2024/11/19-15:18:32.674356 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004611.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:18:32.677639 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004611.sst): index pass time = 0.00's, 156.502'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63577 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:18:32.739573 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004611.sst): first pass time = 0.12's, 564.139'MB/sec 2024/11/19-15:18:32.739630 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004611.sst): old prealloc_size = 73819750, real_size = 64991680 2024/11/19-15:18:32.739690 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004611.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63577 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1017.04 avg-zval =1020.30 2024/11/19-15:18:32.739754 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004611.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.065 sec, Key+Value = 0.066 GB 2024/11/19-15:18:32.798028 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004611.sst): second pass time = 0.00's,129062820.359'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1113.034'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2024.983'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.380'MB/sec write SST data time = 0.02's, 3948.875'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 129062820.359'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63577 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.04 avg-zval = 1020.30 usrkeys = 63577 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.04 avg-zval = 1020.30 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7923 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0453 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 3.792178 GB avg = 0.301 KB (by entry num) total key len = 0.322011 GB avg = 0.026 KB total ukey len = 0.221081 GB avg = 0.018 KB total ukey num = 0.012616248 Billion total entry num = 0.012616248 Billion write speed all = 18.866594536 MB/sec (with seq num) write speed all = 18.403756003 MB/sec (without seq num) 2024/11/19-15:18:32.803120 139851127064320 [compaction_job.cc:1948] [default] [JOB 84] Generated table #4611: 63577 keys, 64977156 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:32.803172 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000712803148, "cf_name": "default", "job": 84, "event": "table_file_creation", "file_number": 4611, "file_size": 64977156, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64867312, "index_size": 108288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1271540, "raw_average_key_size": 20, "raw_value_size": 64660473, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63577, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000712, "oldest_key_time": 0, "file_creation_time": 1732000712, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4611, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:32.817006 139851127064320 [compaction_job.cc:703] job-00084: subcompact[0], size: 0.131115 G, files: /storage/mytopling/.rocksdb [004610,004611] 2024/11/19-15:18:32.817623 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:18:32.817768 139851127064320 (Original Log Time 2024/11/19-15:18:32.426683) EVENT_LOG_v1 {"time_micros": 1732000712426661, "job": 84, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4526, 4527, 4528, 4529], "score": 2.76688, "input_data_size": 134099016, "oldest_snapshot_seqno": -1} 2024/11/19-15:18:32.817777 139851127064320 (Original Log Time 2024/11/19-15:18:32.817424) [compaction_job.cc:2022] [default] [JOB 84] Compacted 4@1 files to L2 => 131114824 bytes 2024/11/19-15:18:32.817782 139851127064320 (Original Log Time 2024/11/19-15:18:32.817593) [compaction_job.cc:1224] [default] compacted to: files[5 41 31 0 0 0 0] max score 2.00 (28 files need compaction), MB/sec: 343.5 rd, 335.9 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128293, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:32.817804 139851127064320 (Original Log Time 2024/11/19-15:18:32.817617) EVENT_LOG_v1 {"time_micros": 1732000712817604, "cf": "default", "job": 84, "event": "compaction_finished", "compaction_time_micros": 390340, "compaction_time_cpu_micros": 387303, "output_level": 2, "num_output_files": 2, "total_output_size": 131114824, "num_input_records": 128293, "num_output_records": 128293, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 41, 31, 0, 0, 0, 0]} 2024/11/19-15:18:32.824447 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004501.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:32.824471 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000712824469, "job": 84, "event": "table_file_deletion", "file_number": 4501} 2024/11/19-15:18:32.831037 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004502.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:32.831059 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000712831056, "job": 84, "event": "table_file_deletion", "file_number": 4502} 2024/11/19-15:18:32.837878 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004507.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:32.837899 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000712837897, "job": 84, "event": "table_file_deletion", "file_number": 4507} 2024/11/19-15:18:32.843869 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004508.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:32.843890 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000712843888, "job": 84, "event": "table_file_deletion", "file_number": 4508} 2024/11/19-15:18:32.844068 139851127064320 [compaction_job.cc:2437] [default] [JOB 100] Compacting 1@1 files to L2, score 0.94, subcompactions 1 : 1 2024/11/19-15:18:32.844077 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 98 Base level 1, inputs: [4567(31MB)] 2024/11/19-15:18:33.043029 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-00 2024/11/19-15:18:33.043115 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00098/att-00/compact.done) = No such file or directory: 3.359 sec on http probe failed: 5, giveup 2024/11/19-15:18:33.043131 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00098/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.699 sec, real 3.667 (submit 297.839 ms wait 3.359 s), input 31.699 MiB, send shutdown 2024/11/19-15:18:33.052129 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-00 2024/11/19-15:18:33.052327 139851137550080 [dcompact_etcd.cc:1773] job-00098/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:33.117252 139851127064320 [dcompact_etcd.cc:1163] job-00100/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.273 sec, input = 31.699 MiB 2024/11/19-15:18:33.181011 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-00 2024/11/19-15:18:33.181063 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00099/att-00/compact.done) = No such file or directory: 2.718 sec on http probe failed: 4, retry 2024/11/19-15:18:33.314146 139851137550080 [dcompact_etcd.cc:1163] job-00098/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.262 sec, input = 31.699 MiB 2024/11/19-15:18:33.317863 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-02 2024/11/19-15:18:33.317913 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00085/att-02/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:18:33.726909 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-01 2024/11/19-15:18:33.727031 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00096/att-01/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:18:33.748489 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-01 2024/11/19-15:18:33.748560 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00094/att-01/compact.done) = No such file or directory: 9.194 sec on http probe failed: 4, retry 2024/11/19-15:18:33.823951 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-00 2024/11/19-15:18:33.824027 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00099/att-00/compact.done) = No such file or directory: 3.357 sec on http probe failed: 5, giveup 2024/11/19-15:18:33.824044 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00099/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.704 sec, real 3.678 (submit 312.233 ms wait 3.357 s), input 31.704 MiB, send shutdown 2024/11/19-15:18:33.831787 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-00 2024/11/19-15:18:33.832010 139851158521600 [dcompact_etcd.cc:1773] job-00099/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:33.918271 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-00 2024/11/19-15:18:33.918375 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00100/att-00/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:18:34.060251 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-01 2024/11/19-15:18:34.060346 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00095/att-01/compact.done) = No such file or directory: 9.200 sec on http probe failed: 4, retry 2024/11/19-15:18:34.096254 139851158521600 [dcompact_etcd.cc:1163] job-00099/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.264 sec, input = 31.704 MiB 2024/11/19-15:18:34.115901 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-01 2024/11/19-15:18:34.115975 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00098/att-01/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:18:34.560522 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-00 2024/11/19-15:18:34.560632 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00100/att-00/compact.done) = No such file or directory: 1.435 sec on http probe failed: 2, retry 2024/11/19-15:18:34.755901 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-01 2024/11/19-15:18:34.755994 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00098/att-01/compact.done) = No such file or directory: 1.436 sec on http probe failed: 2, retry 2024/11/19-15:18:34.898172 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-01 2024/11/19-15:18:34.898251 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00099/att-01/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:18:35.204022 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-00 2024/11/19-15:18:35.204113 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00100/att-00/compact.done) = No such file or directory: 2.077 sec on http probe failed: 3, retry 2024/11/19-15:18:35.326546 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-02 2024/11/19-15:18:35.326620 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00085/att-02/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:18:35.326638 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00085/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.905 sec, real 11.502 (submit 286.618 ms wait 11.207 s), input 126.905 MiB, send shutdown 2024/11/19-15:18:35.334634 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00085/att-02 2024/11/19-15:18:35.334861 139851148035840 [dcompact_etcd.cc:1773] job-00085/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:35.336762 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00085) = success 2024/11/19-15:18:35.336770 139851148035840 [dcompact_etcd.cc:1773] job-00085/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:18:35.336789 139851148035840 (Original Log Time 2024/11/19-15:18:00.783237) EVENT_LOG_v1 {"time_micros": 1732000680783229, "job": 85, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4532, 4533, 4534, 4535], "score": 2.5171, "input_data_size": 134101704, "oldest_snapshot_seqno": 118899982} 2024/11/19-15:18:35.336795 139851148035840 [compaction_job.cc:2437] [default] [JOB 85] Compacting 4@1 files to L2, score 2.52, subcompactions 1 : 1 2024/11/19-15:18:35.336803 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 83 Base level 1, inputs: [4532(31MB) 4533(31MB) 4534(31MB) 4535(31MB)] 2024/11/19-15:18:35.337063 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004612.sst) /storage/mytopling/..//tmp/Topling-82038-sSQYHU 2024/11/19-15:18:35.395511 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004612.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:18:35.397989 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-01 2024/11/19-15:18:35.398040 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004612.sst): index pass time = 0.00's, 208.055'MB/sec index type = UintIndex_AllOne usrkeys = 64717 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:18:35.398110 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00098/att-01/compact.done) = No such file or directory: 2.076 sec on http probe failed: 3, retry 2024/11/19-15:18:35.466884 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004612.sst): first pass time = 0.13's, 516.994'MB/sec 2024/11/19-15:18:35.466950 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004612.sst): old prealloc_size = 73819750, real_size = 66042136 2024/11/19-15:18:35.466982 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004612.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64717 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.97 avg-zval =1020.22 2024/11/19-15:18:35.522011 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004612.sst): second pass time = 0.00's,180810678.571'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1199.291'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.276'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.417'MB/sec write SST data time = 0.02's, 4075.020'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 180810678.571'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64717 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.97 avg-zval = 1020.22 usrkeys = 64717 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.97 avg-zval = 1020.22 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.8125 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 3.857993 GB avg = 0.304 KB (by entry num) total key len = 0.323305 GB avg = 0.025 KB total ukey len = 0.221858 GB avg = 0.017 KB total ukey num = 0.012680965 Billion total entry num = 0.012680965 Billion write speed all = 18.937779619 MB/sec (with seq num) write speed all = 18.478306393 MB/sec (without seq num) 2024/11/19-15:18:35.537220 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-01 2024/11/19-15:18:35.537328 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00099/att-01/compact.done) = No such file or directory: 1.436 sec on http probe failed: 2, retry 2024/11/19-15:18:35.621969 139851148035840 [compaction_job.cc:1948] [default] [JOB 85] Generated table #4612: 64717 keys, 66026954 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:35.622028 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715622004, "cf_name": "default", "job": 85, "event": "table_file_creation", "file_number": 4612, "file_size": 66026954, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025632, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294340, "raw_average_key_size": 20, "raw_value_size": 65815087, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64717, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000715, "oldest_key_time": 0, "file_creation_time": 1732000715, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4612, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:35.637025 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004613.sst) /storage/mytopling/..//tmp/Topling-82038-VtS2x1 2024/11/19-15:18:35.690027 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004613.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:18:35.693268 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004613.sst): index pass time = 0.00's, 158.395'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63607 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:18:35.735932 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-01 2024/11/19-15:18:35.736015 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00096/att-01/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:18:35.753477 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-01 2024/11/19-15:18:35.753530 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00094/att-01/compact.done) = No such file or directory: 11.201 sec on http probe failed: 5, giveup 2024/11/19-15:18:35.753546 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00094/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.812 sec, real 11.470 (submit 263.439 ms wait 11.201 s), input 126.812 MiB, send shutdown 2024/11/19-15:18:35.756622 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004613.sst): first pass time = 0.12's, 551.597'MB/sec 2024/11/19-15:18:35.756668 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004613.sst): old prealloc_size = 73819750, real_size = 65019056 2024/11/19-15:18:35.756723 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004613.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63607 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1016.99 avg-zval =1020.24 2024/11/19-15:18:35.756768 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004613.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.066 GB 2024/11/19-15:18:35.760555 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-01 2024/11/19-15:18:35.760770 139851179493120 [dcompact_etcd.cc:1773] job-00094/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:35.815396 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004613.sst): second pass time = 0.00's,249759397.683'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1107.174'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2154.115'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.381'MB/sec write SST data time = 0.02's, 3846.382'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 249759397.683'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63607 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 usrkeys = 63607 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7926 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0445 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 3.922681 GB avg = 0.308 KB (by entry num) total key len = 0.324577 GB avg = 0.025 KB total ukey len = 0.222621 GB avg = 0.017 KB total ukey num = 0.012744572 Billion total entry num = 0.012744572 Billion write speed all = 19.210995073 MB/sec (with seq num) write speed all = 18.749829945 MB/sec (without seq num) 2024/11/19-15:18:35.819433 139851148035840 [compaction_job.cc:1948] [default] [JOB 85] Generated table #4613: 63607 keys, 65004548 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:35.819485 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715819461, "cf_name": "default", "job": 85, "event": "table_file_creation", "file_number": 4613, "file_size": 65004548, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64894624, "index_size": 108352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1272140, "raw_average_key_size": 20, "raw_value_size": 64687684, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63607, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000715, "oldest_key_time": 0, "file_creation_time": 1732000715, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4613, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:35.833765 139851148035840 [compaction_job.cc:703] job-00085: subcompact[0], size: 0.131032 G, files: /storage/mytopling/.rocksdb [004612,004613] 2024/11/19-15:18:35.834367 139851148035840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:18:35.834546 139851148035840 (Original Log Time 2024/11/19-15:18:35.336835) EVENT_LOG_v1 {"time_micros": 1732000715336815, "job": 85, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4532, 4533, 4534, 4535], "score": 2.5171, "input_data_size": 134101704, "oldest_snapshot_seqno": 118899982} 2024/11/19-15:18:35.834552 139851148035840 (Original Log Time 2024/11/19-15:18:35.834169) [compaction_job.cc:2022] [default] [JOB 85] Compacted 4@1 files to L2 => 131031502 bytes 2024/11/19-15:18:35.834555 139851148035840 (Original Log Time 2024/11/19-15:18:35.834337) [compaction_job.cc:1224] [default] compacted to: files[5 37 33 0 0 0 0] max score 2.22 (27 files need compaction), MB/sec: 269.9 rd, 263.7 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128324, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:35.834560 139851148035840 (Original Log Time 2024/11/19-15:18:35.834361) EVENT_LOG_v1 {"time_micros": 1732000715834348, "cf": "default", "job": 85, "event": "compaction_finished", "compaction_time_micros": 496941, "compaction_time_cpu_micros": 395532, "output_level": 2, "num_output_files": 2, "total_output_size": 131031502, "num_input_records": 128324, "num_output_records": 128324, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 37, 33, 0, 0, 0, 0]} 2024/11/19-15:18:35.834857 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003692.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.834874 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715834872, "job": 85, "event": "table_file_deletion", "file_number": 3692} 2024/11/19-15:18:35.835099 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003697.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.835112 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715835110, "job": 85, "event": "table_file_deletion", "file_number": 3697} 2024/11/19-15:18:35.835298 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003703.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.835308 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715835307, "job": 85, "event": "table_file_deletion", "file_number": 3703} 2024/11/19-15:18:35.835815 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003707.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.835825 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715835824, "job": 85, "event": "table_file_deletion", "file_number": 3707} 2024/11/19-15:18:35.835964 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003709.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.835973 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715835971, "job": 85, "event": "table_file_deletion", "file_number": 3709} 2024/11/19-15:18:35.836234 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003713.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.836244 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715836243, "job": 85, "event": "table_file_deletion", "file_number": 3713} 2024/11/19-15:18:35.836656 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003715.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.836668 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715836667, "job": 85, "event": "table_file_deletion", "file_number": 3715} 2024/11/19-15:18:35.836821 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003718.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.836831 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715836830, "job": 85, "event": "table_file_deletion", "file_number": 3718} 2024/11/19-15:18:35.837417 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003720.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.837430 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715837428, "job": 85, "event": "table_file_deletion", "file_number": 3720} 2024/11/19-15:18:35.837969 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003724.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.837982 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715837981, "job": 85, "event": "table_file_deletion", "file_number": 3724} 2024/11/19-15:18:35.838109 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003725.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.838118 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715838116, "job": 85, "event": "table_file_deletion", "file_number": 3725} 2024/11/19-15:18:35.838721 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003727.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.838733 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715838732, "job": 85, "event": "table_file_deletion", "file_number": 3727} 2024/11/19-15:18:35.839028 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003788.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.839040 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715839039, "job": 85, "event": "table_file_deletion", "file_number": 3788} 2024/11/19-15:18:35.839241 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003796.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.839251 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715839250, "job": 85, "event": "table_file_deletion", "file_number": 3796} 2024/11/19-15:18:35.839661 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003798.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.839671 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715839670, "job": 85, "event": "table_file_deletion", "file_number": 3798} 2024/11/19-15:18:35.839935 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003803.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.839946 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715839944, "job": 85, "event": "table_file_deletion", "file_number": 3803} 2024/11/19-15:18:35.840355 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003806.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.840366 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715840365, "job": 85, "event": "table_file_deletion", "file_number": 3806} 2024/11/19-15:18:35.840754 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003812.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.840764 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715840763, "job": 85, "event": "table_file_deletion", "file_number": 3812} 2024/11/19-15:18:35.841457 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003823.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.841469 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715841468, "job": 85, "event": "table_file_deletion", "file_number": 3823} 2024/11/19-15:18:35.841824 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003945.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.841834 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715841833, "job": 85, "event": "table_file_deletion", "file_number": 3945} 2024/11/19-15:18:35.842497 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003959.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.842510 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715842508, "job": 85, "event": "table_file_deletion", "file_number": 3959} 2024/11/19-15:18:35.842992 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003962.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.843006 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715843004, "job": 85, "event": "table_file_deletion", "file_number": 3962} 2024/11/19-15:18:35.843666 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003963.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.843678 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715843676, "job": 85, "event": "table_file_deletion", "file_number": 3963} 2024/11/19-15:18:35.844230 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003964.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.844241 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715844240, "job": 85, "event": "table_file_deletion", "file_number": 3964} 2024/11/19-15:18:35.844656 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003971.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.844667 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715844665, "job": 85, "event": "table_file_deletion", "file_number": 3971} 2024/11/19-15:18:35.845120 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/003973.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.845132 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715845130, "job": 85, "event": "table_file_deletion", "file_number": 3973} 2024/11/19-15:18:35.846214 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-00 2024/11/19-15:18:35.846297 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00100/att-00/compact.done) = No such file or directory: 2.721 sec on http probe failed: 4, retry 2024/11/19-15:18:35.855976 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004049.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.855999 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715855997, "job": 85, "event": "table_file_deletion", "file_number": 4049} 2024/11/19-15:18:35.867292 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004083.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.867316 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715867314, "job": 85, "event": "table_file_deletion", "file_number": 4083} 2024/11/19-15:18:35.869656 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004084.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.869675 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715869673, "job": 85, "event": "table_file_deletion", "file_number": 4084} 2024/11/19-15:18:35.873827 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004089.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.873848 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715873846, "job": 85, "event": "table_file_deletion", "file_number": 4089} 2024/11/19-15:18:35.878139 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004140.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.878158 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715878156, "job": 85, "event": "table_file_deletion", "file_number": 4140} 2024/11/19-15:18:35.880422 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004141.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.880441 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715880439, "job": 85, "event": "table_file_deletion", "file_number": 4141} 2024/11/19-15:18:35.881800 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004142.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.881817 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715881815, "job": 85, "event": "table_file_deletion", "file_number": 4142} 2024/11/19-15:18:35.884474 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004231.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.884492 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715884490, "job": 85, "event": "table_file_deletion", "file_number": 4231} 2024/11/19-15:18:35.896027 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004267.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.896049 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715896047, "job": 85, "event": "table_file_deletion", "file_number": 4267} 2024/11/19-15:18:35.906607 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004271.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.906627 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715906625, "job": 85, "event": "table_file_deletion", "file_number": 4271} 2024/11/19-15:18:35.907944 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004272.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.907969 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715907967, "job": 85, "event": "table_file_deletion", "file_number": 4272} 2024/11/19-15:18:35.912847 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004273.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.912866 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715912864, "job": 85, "event": "table_file_deletion", "file_number": 4273} 2024/11/19-15:18:35.917395 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004274.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.917414 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715917412, "job": 85, "event": "table_file_deletion", "file_number": 4274} 2024/11/19-15:18:35.922463 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004277.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.922483 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715922481, "job": 85, "event": "table_file_deletion", "file_number": 4277} 2024/11/19-15:18:35.924741 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004278.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.924761 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715924759, "job": 85, "event": "table_file_deletion", "file_number": 4278} 2024/11/19-15:18:35.927120 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004279.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.927138 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715927136, "job": 85, "event": "table_file_deletion", "file_number": 4279} 2024/11/19-15:18:35.931188 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004280.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.931207 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715931205, "job": 85, "event": "table_file_deletion", "file_number": 4280} 2024/11/19-15:18:35.936295 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004281.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.936313 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715936312, "job": 85, "event": "table_file_deletion", "file_number": 4281} 2024/11/19-15:18:35.941204 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004282.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.941223 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715941221, "job": 85, "event": "table_file_deletion", "file_number": 4282} 2024/11/19-15:18:35.945538 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004497.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.945556 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715945554, "job": 85, "event": "table_file_deletion", "file_number": 4497} 2024/11/19-15:18:35.949667 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004500.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.949685 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715949684, "job": 85, "event": "table_file_deletion", "file_number": 4500} 2024/11/19-15:18:35.953999 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004505.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.954020 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715954018, "job": 85, "event": "table_file_deletion", "file_number": 4505} 2024/11/19-15:18:35.958352 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004509.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:35.958372 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000715958370, "job": 85, "event": "table_file_deletion", "file_number": 4509} 2024/11/19-15:18:35.958540 139851148035840 [compaction_job.cc:2437] [default] [JOB 101] Compacting 4@2 files to L3, score 1.02, subcompactions 1 : 1 2024/11/19-15:18:35.958548 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 99 Base level 2, inputs: [4530(10KB) 4531(2105B) 4575(3258B) 4576(29KB)] 2024/11/19-15:18:36.037380 139851179493120 [dcompact_etcd.cc:1163] job-00094/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.277 sec, input = 126.812 MiB 2024/11/19-15:18:36.042307 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-01 2024/11/19-15:18:36.042364 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00098/att-01/compact.done) = No such file or directory: 2.718 sec on http probe failed: 4, retry 2024/11/19-15:18:36.065049 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-01 2024/11/19-15:18:36.065102 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00095/att-01/compact.done) = No such file or directory: 11.209 sec on http probe failed: 5, giveup 2024/11/19-15:18:36.065119 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00095/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.833 sec, real 11.494 (submit 280.159 ms wait 11.209 s), input 126.833 MiB, send shutdown 2024/11/19-15:18:36.074752 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-01 2024/11/19-15:18:36.074971 139850978166528 [dcompact_etcd.cc:1773] job-00095/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:36.180016 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-01 2024/11/19-15:18:36.180087 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00099/att-01/compact.done) = No such file or directory: 2.075 sec on http probe failed: 3, retry 2024/11/19-15:18:36.256863 139851148035840 [dcompact_etcd.cc:1163] job-00101/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.298 sec, input = 24.491 KiB 2024/11/19-15:18:36.266411 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-00 2024/11/19-15:18:36.266498 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00101/att-00/compact.done) = No such file or directory: 0.001 sec on http probe failed: 1, retry 2024/11/19-15:18:36.272600 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-00 2024/11/19-15:18:36.272663 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00101/att-00/compact.done) = No such file or directory: 0.010 sec on http probe failed: 2, retry 2024/11/19-15:18:36.282739 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-00 2024/11/19-15:18:36.282798 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00101/att-00/compact.done) = No such file or directory: 0.016 sec on http probe failed: 3, retry 2024/11/19-15:18:36.291807 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-00 2024/11/19-15:18:36.291870 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00101/att-00/compact.done) = No such file or directory: 0.026 sec on http probe failed: 4, retry 2024/11/19-15:18:36.301518 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-00 2024/11/19-15:18:36.301578 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00101/att-00/compact.done) = No such file or directory: 0.036 sec on http probe failed: 5, giveup 2024/11/19-15:18:36.301599 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00101/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.343 (submit 297.794 ms wait 0.036 s), input 24.491 KiB, send shutdown 2024/11/19-15:18:36.306219 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-00 2024/11/19-15:18:36.306459 139851148035840 [dcompact_etcd.cc:1773] job-00101/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:36.344862 139850978166528 [dcompact_etcd.cc:1163] job-00095/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.270 sec, input = 126.833 MiB 2024/11/19-15:18:36.486157 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-00 2024/11/19-15:18:36.486228 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00100/att-00/compact.done) = No such file or directory: 3.363 sec on http probe failed: 5, giveup 2024/11/19-15:18:36.486244 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00100/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.699 sec, real 3.642 (submit 272.755 ms wait 3.363 s), input 31.699 MiB, send shutdown 2024/11/19-15:18:36.492203 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-00 2024/11/19-15:18:36.492421 139851127064320 [dcompact_etcd.cc:1773] job-00100/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:36.592694 139851148035840 [dcompact_etcd.cc:1163] job-00101/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.286 sec, input = 24.491 KiB 2024/11/19-15:18:36.602453 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-01 2024/11/19-15:18:36.602520 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00101/att-01/compact.done) = No such file or directory: 0.001 sec on http probe failed: 1, retry 2024/11/19-15:18:36.607638 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-01 2024/11/19-15:18:36.607681 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00101/att-01/compact.done) = No such file or directory: 0.010 sec on http probe failed: 2, retry 2024/11/19-15:18:36.616201 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-01 2024/11/19-15:18:36.616245 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00101/att-01/compact.done) = No such file or directory: 0.016 sec on http probe failed: 3, retry 2024/11/19-15:18:36.624362 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-01 2024/11/19-15:18:36.624403 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00101/att-01/compact.done) = No such file or directory: 0.024 sec on http probe failed: 4, retry 2024/11/19-15:18:36.634152 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-01 2024/11/19-15:18:36.634192 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00101/att-01/compact.done) = No such file or directory: 0.032 sec on http probe failed: 5, giveup 2024/11/19-15:18:36.634205 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00101/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.328 (submit 286.140 ms wait 0.032 s), input 24.491 KiB, send shutdown 2024/11/19-15:18:36.638954 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-01 2024/11/19-15:18:36.639155 139851148035840 [dcompact_etcd.cc:1773] job-00101/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:36.684082 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-01 2024/11/19-15:18:36.684135 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00098/att-01/compact.done) = No such file or directory: 3.362 sec on http probe failed: 5, giveup 2024/11/19-15:18:36.684149 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00098/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.699 sec, real 3.632 (submit 261.739 ms wait 3.362 s), input 31.699 MiB, send shutdown 2024/11/19-15:18:36.692927 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-01 2024/11/19-15:18:36.693129 139851137550080 [dcompact_etcd.cc:1773] job-00098/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:36.795559 139851127064320 [dcompact_etcd.cc:1163] job-00100/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.303 sec, input = 31.699 MiB 2024/11/19-15:18:36.797182 139852362282752 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16273899 2024/11/19-15:18:36.822525 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-01 2024/11/19-15:18:36.822589 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00099/att-01/compact.done) = No such file or directory: 2.718 sec on http probe failed: 4, retry 2024/11/19-15:18:36.919859 139851148035840 [dcompact_etcd.cc:1163] job-00101/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 24.491 KiB 2024/11/19-15:18:36.929910 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-02 2024/11/19-15:18:36.929986 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00101/att-02/compact.done) = No such file or directory: 0.001 sec on http probe failed: 1, retry 2024/11/19-15:18:36.940177 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-02 2024/11/19-15:18:36.940222 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00101/att-02/compact.done) = No such file or directory: 0.011 sec on http probe failed: 2, retry 2024/11/19-15:18:36.941456 139851137550080 [dcompact_etcd.cc:1163] job-00098/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.248 sec, input = 31.699 MiB 2024/11/19-15:18:36.945544 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-02 2024/11/19-15:18:36.945587 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00101/att-02/compact.done) = No such file or directory: 0.021 sec on http probe failed: 3, retry 2024/11/19-15:18:36.952928 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-02 2024/11/19-15:18:36.952986 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00101/att-02/compact.done) = No such file or directory: 0.026 sec on http probe failed: 4, retry 2024/11/19-15:18:36.960797 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-02 2024/11/19-15:18:36.960840 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00101/att-02/compact.done) = No such file or directory: 0.034 sec on http probe failed: 5, giveup 2024/11/19-15:18:36.960854 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00101/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.322 (submit 280.637 ms wait 0.034 s), input 24.491 KiB, send shutdown 2024/11/19-15:18:36.965908 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00101/att-02 2024/11/19-15:18:36.966132 139851148035840 [dcompact_etcd.cc:1773] job-00101/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:36.967903 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00101) = success 2024/11/19-15:18:36.967912 139851148035840 [dcompact_etcd.cc:1773] job-00101/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:18:36.967944 139851148035840 (Original Log Time 2024/11/19-15:18:35.958563) EVENT_LOG_v1 {"time_micros": 1732000715958554, "job": 101, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4530, 4531, 4575, 4576], "score": 1.02434, "input_data_size": 46320, "oldest_snapshot_seqno": 121231580} 2024/11/19-15:18:36.967952 139851148035840 [compaction_job.cc:2437] [default] [JOB 101] Compacting 4@2 files to L3, score 1.02, subcompactions 1 : 1 2024/11/19-15:18:36.967960 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 99 Base level 2, inputs: [4530(10KB) 4531(2105B) 4575(3258B) 4576(29KB)] 2024/11/19-15:18:36.968197 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004614.sst) /storage/mytopling/..//tmp/Topling-82038-pgWUF5 2024/11/19-15:18:36.968318 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004614.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:18:36.968477 139834045761280 [fixed_len_key_index.cc:363] FixedLenKeyIndex build_cache Finish: num_keys = 4, fixlen = 27, pref_len = 4, cplen = 0, time = 0.000 sec 2024/11/19-15:18:36.968536 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004614.sst): first pass time = 0.00's, 0.715'MB/sec 2024/11/19-15:18:36.968557 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004614.sst): index pass time = 0.00's, 0.493'MB/sec index type = FixedLenKeyIndex usrkeys = 4 min-keylen = 27 max-keylen = 27 prefix = 4 subcplen = 0 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 27.00 avg-zkey =2256.00 2024/11/19-15:18:36.968606 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004614.sst): old prealloc_size = 50952, real_size = 9568 2024/11/19-15:18:36.968622 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004614.sst): index type = FixedLenKeyIndex , store type = ZipOffsetBlobStore usrkeys = 4 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 27.00 avg-zkey =2256.00 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 20.00 avg-zval = 84.00 2024/11/19-15:18:36.968761 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004614.sst): second pass time = 0.00's, 625.000'MB/sec, value only(33.9% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 72.786'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 539.423'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.096'MB/sec write SST data time = 0.00's, 88.952'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 625.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 4 avg-key = 39.00 avg-zkey = 2256.00 avg-val = 20.00 avg-zval = 84.00 usrkeys = 4 avg-key = 31.00 avg-zkey = 2256.00 avg-val = 20.00 avg-zval = 84.00 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 3 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.0137 value = 0.2381 dict = 0.00 all = 0.0225 } Zip/UnZip{ index = 72.7742 value = 4.2000 dict = 0.00 all = 44.4492 } ---------------------------- total value len = 3.922681 GB avg = 0.308 KB (by entry num) total key len = 0.324578 GB avg = 0.025 KB total ukey len = 0.222621 GB avg = 0.017 KB total ukey num = 0.012744576 Billion total entry num = 0.012744576 Billion write speed all =7059209.722402278 MB/sec (with seq num) write speed all =6889751.441839438 MB/sec (without seq num) 2024/11/19-15:18:36.968987 139851148035840 [compaction_job.cc:1948] [default] [JOB 101] Generated table #4614: 4 keys, 10490 bytes, temperature: kUnknown 2024/11/19-15:18:36.969020 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000716969001, "cf_name": "default", "job": 101, "event": "table_file_creation", "file_number": 4614, "file_size": 10490, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 336, "index_size": 9024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 156, "raw_average_key_size": 39, "raw_value_size": 80, "raw_average_value_size": 20, "num_data_blocks": 1, "num_entries": 4, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 39, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-FixedLenKeyIndex-ZipOffsetBlobStore", "creation_time": 1732000716, "oldest_key_time": 0, "file_creation_time": 1732000716, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4614, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:36.969170 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004615.sst) /storage/mytopling/..//tmp/Topling-82038-bqb9N9 2024/11/19-15:18:36.969276 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004615.sst): first pass time = 0.00's, 23.782'MB/sec 2024/11/19-15:18:36.969285 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004615.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:18:36.969360 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004615.sst): index pass time = 0.00's, 1.314'MB/sec index type = UintIndex_AllOne usrkeys = 18 min-keylen = 4 max-keylen = 4 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 8.89 2024/11/19-15:18:36.969627 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004615.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, dictZip workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:18:36.990066 139851148035840 [top_zip_table_builder.cc:2752] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004615.sst): part-0: reject ZipStore ratio = 0.881 2024/11/19-15:18:36.990262 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004615.sst): old prealloc_size = 50952, real_size = 2720 2024/11/19-15:18:36.990277 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004615.sst): index type = UintIndex_AllOne , store type = ZipOffsetBlobStore usrkeys = 18 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 8.89 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 115.06 avg-zval = 130.67 2024/11/19-15:18:36.990400 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004615.sst): second pass time = 0.02's, 0.100'MB/sec, value only(87.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 28.459'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 11.913'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.862'MB/sec write SST data time = 0.00's, 33.835'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = 0.000'MB/sec zip my value time = 0.02's, unzip length = 0.000'GB zip my value throughput = 0.100'MB/sec zip pipeline throughput = 0.100'MB/sec entries = 18 avg-key = 16.00 avg-zkey = 10.67 avg-val = 115.06 avg-zval = 130.67 usrkeys = 18 avg-key = 8.00 avg-zkey = 10.67 avg-val = 115.06 avg-zval = 130.67 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 4 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.7500 value = 0.8805 dict = 0.00 all = 0.6475 } Zip/UnZip{ index = 1.3333 value = 1.1357 dict = 0.00 all = 1.5443 } ---------------------------- total value len = 3.922683 GB avg = 0.308 KB (by entry num) total key len = 0.324578 GB avg = 0.025 KB total ukey len = 0.222621 GB avg = 0.017 KB total ukey num = 0.012744594 Billion total entry num = 0.012744594 Billion write speed all = 190968.117430228 MB/sec (with seq num) write speed all = 186383.871058973 MB/sec (without seq num) 2024/11/19-15:18:36.990709 139851148035840 [compaction_job.cc:1948] [default] [JOB 101] Generated table #4615: 18 keys, 3643 bytes, temperature: kUnknown 2024/11/19-15:18:36.990740 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000716990722, "cf_name": "default", "job": 101, "event": "table_file_creation", "file_number": 4615, "file_size": 3643, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 2352, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 288, "raw_average_key_size": 16, "raw_value_size": 2071, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 18, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-ZipOffsetBlobStore", "creation_time": 1732000716, "oldest_key_time": 0, "file_creation_time": 1732000716, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4615, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:36.990878 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004616.sst) /storage/mytopling/..//tmp/Topling-82038-DnRQOL 2024/11/19-15:18:36.991040 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004616.sst): first pass time = 0.00's, 144.209'MB/sec 2024/11/19-15:18:36.991043 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004616.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:18:36.991066 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004616.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, dictZip workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:18:36.991172 139834024789760 [fixed_len_key_index.cc:363] FixedLenKeyIndex build_cache Finish: num_keys = 160, fixlen = 8, pref_len = 4, cplen = 3, time = 0.000 sec 2024/11/19-15:18:36.991233 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004616.sst): index pass time = 0.00's, 7.538'MB/sec index type = FixedLenKeyIndex usrkeys = 160 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 59.20 2024/11/19-15:18:37.011580 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004616.sst): old prealloc_size = 50952, real_size = 23104 2024/11/19-15:18:37.011604 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004616.sst): index type = FixedLenKeyIndex , store type = DictZipBlobStore usrkeys = 160 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 59.20 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 120.53 avg-zval = 83.90 2024/11/19-15:18:37.011778 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004616.sst): second pass time = 0.02's, 0.940'MB/sec, value only(85.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 135.433'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 531.866'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 6.811'MB/sec write SST data time = 0.00's, 154.932'MB/sec dict compress time = 0.00's, 6.141'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = 5.629'MB/sec zip my value time = 0.02's, unzip length = 0.000'GB zip my value throughput = 0.940'MB/sec zip pipeline throughput = 0.940'MB/sec entries = 160 avg-key = 20.00 avg-zkey = 59.20 avg-val = 120.53 avg-zval = 83.90 usrkeys = 160 avg-key = 12.00 avg-zkey = 59.20 avg-val = 120.53 avg-zval = 83.90 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 3 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.2027 value = 1.4365 dict = 1.00 all = 0.9211 } Zip/UnZip{ index = 4.9333 value = 0.6961 dict = 1.00 all = 1.0856 } ---------------------------- total value len = 3.922702 GB avg = 0.308 KB (by entry num) total key len = 0.324581 GB avg = 0.025 KB total ukey len = 0.222623 GB avg = 0.017 KB total ukey num = 0.012744754 Billion total entry num = 0.012744754 Billion write speed all = 97384.040133766 MB/sec (with seq num) write speed all = 95046.290356740 MB/sec (without seq num) 2024/11/19-15:18:37.012121 139851148035840 [compaction_job.cc:1948] [default] [JOB 101] Generated table #4616: 160 keys, 24409 bytes, temperature: kUnknown 2024/11/19-15:18:37.012168 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000717012141, "cf_name": "default", "job": 101, "event": "table_file_creation", "file_number": 4616, "file_size": 24409, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 13424, "index_size": 9472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3200, "raw_average_key_size": 20, "raw_value_size": 19284, "raw_average_value_size": 120, "num_data_blocks": 1, "num_entries": 160, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-FixedLenKeyIndex-DictZipBlobStore", "creation_time": 1732000716, "oldest_key_time": 0, "file_creation_time": 1732000716, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4616, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:37.012291 139851148035840 [compaction_job.cc:703] job-00101: subcompact[0], size: 0.000039 G, files: /storage/mytopling/.rocksdb [004614,004615,004616] 2024/11/19-15:18:37.013014 139851148035840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 15785682 2024/11/19-15:18:37.013166 139851148035840 (Original Log Time 2024/11/19-15:18:36.967990) EVENT_LOG_v1 {"time_micros": 1732000716967973, "job": 101, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4530, 4531, 4575, 4576], "score": 1.02434, "input_data_size": 46320, "oldest_snapshot_seqno": 121231580} 2024/11/19-15:18:37.013175 139851148035840 (Original Log Time 2024/11/19-15:18:37.012742) [compaction_job.cc:2022] [default] [JOB 101] Compacted 4@2 files to L3 => 38542 bytes 2024/11/19-15:18:37.013181 139851148035840 (Original Log Time 2024/11/19-15:18:37.012982) [compaction_job.cc:1224] [default] compacted to: files[5 37 29 3 0 0 0] max score 2.22 (56 files need compaction), MB/sec: 1.0 rd, 0.9 wr, level 3, files in(4, 0) out(3 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 182, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:37.013198 139851148035840 (Original Log Time 2024/11/19-15:18:37.013007) EVENT_LOG_v1 {"time_micros": 1732000717012993, "cf": "default", "job": 101, "event": "compaction_finished", "compaction_time_micros": 44310, "compaction_time_cpu_micros": 3553, "output_level": 3, "num_output_files": 3, "total_output_size": 38542, "num_input_records": 182, "num_output_records": 182, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 37, 29, 3, 0, 0, 0]} 2024/11/19-15:18:37.013293 139851148035840 [compaction_job.cc:2437] [default] [JOB 102] Compacting 4@2 files to L3, score 1.02, subcompactions 1 : 1 2024/11/19-15:18:37.013304 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 100 Base level 2, inputs: [4577(61MB) 4580(62MB) 4582(61MB) 4587(62MB)] 2024/11/19-15:18:37.274707 139851148035840 [dcompact_etcd.cc:1163] job-00102/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.261 sec, input = 254.856 MiB 2024/11/19-15:18:37.461495 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-01 2024/11/19-15:18:37.461578 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00099/att-01/compact.done) = No such file or directory: 3.360 sec on http probe failed: 5, giveup 2024/11/19-15:18:37.461597 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00099/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.704 sec, real 3.630 (submit 264.168 ms wait 3.360 s), input 31.704 MiB, send shutdown 2024/11/19-15:18:37.466152 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-01 2024/11/19-15:18:37.466374 139851158521600 [dcompact_etcd.cc:1773] job-00099/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:37.597279 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-01 2024/11/19-15:18:37.597358 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00100/att-01/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:18:37.741223 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-02 2024/11/19-15:18:37.741322 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00098/att-02/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:18:37.743659 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-01 2024/11/19-15:18:37.743722 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00096/att-01/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:18:37.743738 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00096/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.961 sec, real 11.499 (submit 283.749 ms wait 11.208 s), input 126.961 MiB, send shutdown 2024/11/19-15:18:37.748072 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-01 2024/11/19-15:18:37.748279 139851169007360 [dcompact_etcd.cc:1773] job-00096/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:37.749581 139851158521600 [dcompact_etcd.cc:1163] job-00099/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.283 sec, input = 31.704 MiB 2024/11/19-15:18:38.011349 139851169007360 [dcompact_etcd.cc:1163] job-00096/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.263 sec, input = 126.961 MiB 2024/11/19-15:18:38.239793 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-01 2024/11/19-15:18:38.239872 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00100/att-01/compact.done) = No such file or directory: 1.436 sec on http probe failed: 2, retry 2024/11/19-15:18:38.383768 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-02 2024/11/19-15:18:38.383863 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00098/att-02/compact.done) = No such file or directory: 1.434 sec on http probe failed: 2, retry 2024/11/19-15:18:38.550730 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-02 2024/11/19-15:18:38.550816 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00099/att-02/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:18:38.878552 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-01 2024/11/19-15:18:38.878645 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00100/att-01/compact.done) = No such file or directory: 2.078 sec on http probe failed: 3, retry 2024/11/19-15:18:39.022692 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-02 2024/11/19-15:18:39.022771 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00098/att-02/compact.done) = No such file or directory: 2.076 sec on http probe failed: 3, retry 2024/11/19-15:18:39.193542 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-02 2024/11/19-15:18:39.193644 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00099/att-02/compact.done) = No such file or directory: 1.435 sec on http probe failed: 2, retry 2024/11/19-15:18:39.215405 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-02 2024/11/19-15:18:39.215497 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00094/att-02/compact.done) = No such file or directory: 3.170 sec on http probe failed: 1, retry 2024/11/19-15:18:39.520873 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-01 2024/11/19-15:18:39.520965 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00100/att-01/compact.done) = No such file or directory: 2.717 sec on http probe failed: 4, retry 2024/11/19-15:18:39.523994 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-02 2024/11/19-15:18:39.524054 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00095/att-02/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:18:39.663104 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-02 2024/11/19-15:18:39.663178 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00098/att-02/compact.done) = No such file or directory: 2.715 sec on http probe failed: 4, retry 2024/11/19-15:18:39.837822 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-02 2024/11/19-15:18:39.837915 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00099/att-02/compact.done) = No such file or directory: 2.078 sec on http probe failed: 3, retry 2024/11/19-15:18:40.161320 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-01 2024/11/19-15:18:40.161398 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00100/att-01/compact.done) = No such file or directory: 3.359 sec on http probe failed: 5, giveup 2024/11/19-15:18:40.161417 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00100/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.699 sec, real 3.669 (submit 303.042 ms wait 3.359 s), input 31.699 MiB, send shutdown 2024/11/19-15:18:40.169695 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-01 2024/11/19-15:18:40.169912 139851127064320 [dcompact_etcd.cc:1773] job-00100/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:40.304138 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-02 2024/11/19-15:18:40.304229 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00098/att-02/compact.done) = No such file or directory: 3.356 sec on http probe failed: 5, giveup 2024/11/19-15:18:40.304247 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00098/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.699 sec, real 3.611 (submit 248.244 ms wait 3.356 s), input 31.699 MiB, send shutdown 2024/11/19-15:18:40.312441 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00098/att-02 2024/11/19-15:18:40.327080 139851137550080 [dcompact_etcd.cc:1773] job-00098/att-02: CleanFiles 0.015 sec : 0.000 2024/11/19-15:18:40.357079 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00098) = success 2024/11/19-15:18:40.357090 139851137550080 [dcompact_etcd.cc:1773] job-00098/att-03: CleanFiles 0.030 sec : 0.000 2024/11/19-15:18:40.357110 139851137550080 (Original Log Time 2024/11/19-15:18:29.376042) EVENT_LOG_v1 {"time_micros": 1732000709376034, "job": 98, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [4564], "score": 0.776497, "input_data_size": 33524754, "oldest_snapshot_seqno": 120862495} 2024/11/19-15:18:40.357117 139851137550080 [compaction_job.cc:2437] [default] [JOB 98] Compacting 1@1 files to L2, score 0.78, subcompactions 1 : 1 2024/11/19-15:18:40.357124 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 96 Base level 1, inputs: [4564(31MB)] 2024/11/19-15:18:40.357380 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004617.sst) /storage/mytopling/..//tmp/Topling-82038-ZVgxx3 2024/11/19-15:18:40.384893 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004617.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.033 GB 2024/11/19-15:18:40.385033 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004617.sst): index pass time = 0.00's,2429.645'MB/sec index type = UintIndex_AllOne usrkeys = 32054 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0003 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.01 2024/11/19-15:18:40.426035 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004617.sst): first pass time = 0.07's, 484.270'MB/sec 2024/11/19-15:18:40.426125 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004617.sst): old prealloc_size = 36877229, real_size = 32706456 2024/11/19-15:18:40.426150 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004617.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 32054 prefix = 4 raw-key = 0.0003 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.01 raw-val = 0.0326 GB zip-val = 0.0327 GB avg-val =1016.96 avg-zval =1020.09 2024/11/19-15:18:40.442776 139851127064320 [dcompact_etcd.cc:1163] job-00100/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.273 sec, input = 31.699 MiB 2024/11/19-15:18:40.461460 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004617.sst): second pass time = 0.00's,69062889.831'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.04's, 926.431'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.781'MB/sec (index lex order gen) rebuild zvType time = 0.03's, 0.311'MB/sec write SST data time = 0.01's, 3437.795'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.033'GB zip my value throughput = 69062889.831'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 32054 avg-key = 20.00 avg-zkey = 0.01 avg-val = 1016.96 avg-zval = 1020.09 usrkeys = 32054 avg-key = 12.00 avg-zkey = 0.01 avg-val = 1016.96 avg-zval = 1020.09 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0004 GB value = 0.0326 GB dict = 0.00 MB all = 0.0332 GB } __ZipSize{ index = 0.0000 GB value = 0.0327 GB dict = 0.00 MB all = 0.0327 GB } UnZip/Zip{ index =2003.3750 value = 0.9969 dict = 0.00 all = 1.0165 } Zip/UnZip{ index = 0.0005 value = 1.0031 dict = 0.00 all = 0.9838 } ---------------------------- total value len = 3.955300 GB avg = 0.310 KB (by entry num) total key len = 0.325222 GB avg = 0.025 KB total ukey len = 0.223008 GB avg = 0.017 KB total ukey num = 0.012776808 Billion total entry num = 0.012776808 Billion write speed all = 1225.364764932 MB/sec (with seq num) write speed all = 1196.104313256 MB/sec (without seq num) 2024/11/19-15:18:40.476654 139851137550080 [compaction_job.cc:1948] [default] [JOB 98] Generated table #4617: 32054 keys, 32699402 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:40.476704 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000720476681, "cf_name": "default", "job": 98, "event": "table_file_creation", "file_number": 4617, "file_size": 32699402, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32698080, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641080, "raw_average_key_size": 20, "raw_value_size": 32597684, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000720, "oldest_key_time": 0, "file_creation_time": 1732000720, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4617, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:40.481295 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-02 2024/11/19-15:18:40.481361 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00099/att-02/compact.done) = No such file or directory: 2.723 sec on http probe failed: 4, retry 2024/11/19-15:18:40.485204 139851137550080 [compaction_job.cc:703] job-00098: subcompact[0], size: 0.032699 G, files: /storage/mytopling/.rocksdb [004617] 2024/11/19-15:18:40.488241 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16273898 2024/11/19-15:18:40.488462 139851137550080 (Original Log Time 2024/11/19-15:18:40.357157) EVENT_LOG_v1 {"time_micros": 1732000720357136, "job": 98, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [4564], "score": 0.776497, "input_data_size": 33524754, "oldest_snapshot_seqno": 120862495} 2024/11/19-15:18:40.488482 139851137550080 (Original Log Time 2024/11/19-15:18:40.487864) [compaction_job.cc:2022] [default] [JOB 98] Compacted 1@1 files to L2 => 32699402 bytes 2024/11/19-15:18:40.488492 139851137550080 (Original Log Time 2024/11/19-15:18:40.488186) [compaction_job.cc:1224] [default] compacted to: files[5 36 30 3 0 0 0] max score 2.28 (53 files need compaction), MB/sec: 261.8 rd, 255.3 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(32.0, 0.0 +0.0 blob) out(31.2 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 32054, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:40.488537 139851137550080 (Original Log Time 2024/11/19-15:18:40.488228) EVENT_LOG_v1 {"time_micros": 1732000720488208, "cf": "default", "job": 98, "event": "compaction_finished", "compaction_time_micros": 128072, "compaction_time_cpu_micros": 110005, "output_level": 2, "num_output_files": 1, "total_output_size": 32699402, "num_input_records": 32054, "num_output_records": 32054, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 36, 30, 3, 0, 0, 0]} 2024/11/19-15:18:40.488692 139851137550080 [compaction_job.cc:2437] [default] [JOB 103] Compacting 1@1 files to L2, score 0.88, subcompactions 1 : 1 2024/11/19-15:18:40.488707 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 101 Base level 1, inputs: [4554(31MB)] 2024/11/19-15:18:40.843492 139851137550080 [dcompact_etcd.cc:1163] job-00103/att-00: init time wrt = 0.001, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.354 sec, input = 31.795 MiB 2024/11/19-15:18:41.123350 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-02 2024/11/19-15:18:41.123441 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00099/att-02/compact.done) = No such file or directory: 3.366 sec on http probe failed: 5, giveup 2024/11/19-15:18:41.123460 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00099/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.704 sec, real 3.657 (submit 283.136 ms wait 3.366 s), input 31.704 MiB, send shutdown 2024/11/19-15:18:41.132297 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00099/att-02 2024/11/19-15:18:41.180870 139851158521600 [dcompact_etcd.cc:1773] job-00099/att-02: CleanFiles 0.049 sec : 0.000 2024/11/19-15:18:41.182817 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00099) = success 2024/11/19-15:18:41.182828 139851158521600 [dcompact_etcd.cc:1773] job-00099/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:18:41.182867 139851158521600 (Original Log Time 2024/11/19-15:18:30.145349) EVENT_LOG_v1 {"time_micros": 1732000710145342, "job": 99, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [4552], "score": 0.859119, "input_data_size": 33525330, "oldest_snapshot_seqno": 120902241} 2024/11/19-15:18:41.182875 139851158521600 [compaction_job.cc:2437] [default] [JOB 99] Compacting 1@1 files to L2, score 0.86, subcompactions 1 : 1 2024/11/19-15:18:41.182882 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 97 Base level 1, inputs: [4552(31MB)] 2024/11/19-15:18:41.183136 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004618.sst) /storage/mytopling/..//tmp/Topling-82038-VjzBET 2024/11/19-15:18:41.190195 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-02 2024/11/19-15:18:41.190275 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00096/att-02/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:18:41.210008 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004618.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.033 GB 2024/11/19-15:18:41.210115 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004618.sst): index pass time = 0.00's,2834.357'MB/sec index type = UintIndex_AllOne usrkeys = 32058 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0003 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.01 2024/11/19-15:18:41.220754 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-02 2024/11/19-15:18:41.220829 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00094/att-02/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:18:41.240028 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-02 2024/11/19-15:18:41.240101 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00100/att-02/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:18:41.243179 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004618.sst): first pass time = 0.06's, 553.786'MB/sec 2024/11/19-15:18:41.243221 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004618.sst): old prealloc_size = 36877863, real_size = 32712088 2024/11/19-15:18:41.243243 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004618.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 32058 prefix = 4 raw-key = 0.0003 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.01 raw-val = 0.0326 GB zip-val = 0.0327 GB avg-val =1017.01 avg-zval =1020.14 2024/11/19-15:18:41.269759 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004618.sst): second pass time = 0.00's,91582362.360'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.03's, 1233.184'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.052'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 0.427'MB/sec write SST data time = 0.01's, 4221.323'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.033'GB zip my value throughput = 91582362.360'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 32058 avg-key = 20.00 avg-zkey = 0.01 avg-val = 1017.01 avg-zval = 1020.14 usrkeys = 32058 avg-key = 12.00 avg-zkey = 0.01 avg-val = 1017.01 avg-zval = 1020.14 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0004 GB value = 0.0326 GB dict = 0.00 MB all = 0.0332 GB } __ZipSize{ index = 0.0000 GB value = 0.0327 GB dict = 0.00 MB all = 0.0327 GB } UnZip/Zip{ index =2003.6250 value = 0.9969 dict = 0.00 all = 1.0165 } Zip/UnZip{ index = 0.0005 value = 1.0031 dict = 0.00 all = 0.9838 } ---------------------------- total value len = 3.987903 GB avg = 0.311 KB (by entry num) total key len = 0.325863 GB avg = 0.025 KB total ukey len = 0.223392 GB avg = 0.017 KB total ukey num = 0.012808866 Billion total entry num = 0.012808866 Billion write speed all = 1002.830965699 MB/sec (with seq num) write speed all = 979.009318473 MB/sec (without seq num) 2024/11/19-15:18:41.271730 139851158521600 [compaction_job.cc:1948] [default] [JOB 99] Generated table #4618: 32058 keys, 32705034 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:41.271783 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000721271758, "cf_name": "default", "job": 99, "event": "table_file_creation", "file_number": 4618, "file_size": 32705034, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32703712, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603321, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000721, "oldest_key_time": 0, "file_creation_time": 1732000721, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4618, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:41.278894 139851158521600 [compaction_job.cc:703] job-00099: subcompact[0], size: 0.032705 G, files: /storage/mytopling/.rocksdb [004618] 2024/11/19-15:18:41.279355 139851158521600 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777214 2024/11/19-15:18:41.279461 139851158521600 (Original Log Time 2024/11/19-15:18:41.182911) EVENT_LOG_v1 {"time_micros": 1732000721182895, "job": 99, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [4552], "score": 0.859119, "input_data_size": 33525330, "oldest_snapshot_seqno": 120902241} 2024/11/19-15:18:41.279467 139851158521600 (Original Log Time 2024/11/19-15:18:41.279158) [compaction_job.cc:2022] [default] [JOB 99] Compacted 1@1 files to L2 => 32705034 bytes 2024/11/19-15:18:41.279470 139851158521600 (Original Log Time 2024/11/19-15:18:41.279327) [compaction_job.cc:1224] [default] compacted to: files[5 35 31 3 0 0 0] max score 2.35 (53 files need compaction), MB/sec: 349.2 rd, 340.7 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(32.0, 0.0 +0.0 blob) out(31.2 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 32058, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:41.279482 139851158521600 (Original Log Time 2024/11/19-15:18:41.279349) EVENT_LOG_v1 {"time_micros": 1732000721279337, "cf": "default", "job": 99, "event": "compaction_finished", "compaction_time_micros": 95995, "compaction_time_cpu_micros": 95453, "output_level": 2, "num_output_files": 1, "total_output_size": 32705034, "num_input_records": 32058, "num_output_records": 32058, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 35, 31, 3, 0, 0, 0]} 2024/11/19-15:18:41.279548 139851158521600 [compaction_job.cc:2437] [default] [JOB 104] Compacting 1@2 files to L3, score 0.90, subcompactions 1 : 1 2024/11/19-15:18:41.279556 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 102 Base level 2, inputs: [4588(49MB)] 2024/11/19-15:18:41.536756 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-02 2024/11/19-15:18:41.536827 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00095/att-02/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:18:41.575563 139851158521600 [dcompact_etcd.cc:1163] job-00104/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.296 sec, input = 51.378 MiB 2024/11/19-15:18:41.643742 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-00 2024/11/19-15:18:41.643807 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00103/att-00/compact.done) = No such file or directory: 0.795 sec on http probe failed: 1, retry 2024/11/19-15:18:41.879021 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-02 2024/11/19-15:18:41.879087 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00100/att-02/compact.done) = No such file or directory: 1.431 sec on http probe failed: 2, retry 2024/11/19-15:18:42.285012 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-00 2024/11/19-15:18:42.285125 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00102/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:18:42.288516 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-00 2024/11/19-15:18:42.288598 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00103/att-00/compact.done) = No such file or directory: 1.436 sec on http probe failed: 2, retry 2024/11/19-15:18:42.521874 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-02 2024/11/19-15:18:42.521995 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00100/att-02/compact.done) = No such file or directory: 2.070 sec on http probe failed: 3, retry 2024/11/19-15:18:42.869572 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-00 2024/11/19-15:18:42.869656 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00104/att-00/compact.done) = No such file or directory: 1.285 sec on http probe failed: 1, retry 2024/11/19-15:18:42.931533 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-00 2024/11/19-15:18:42.931599 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00103/att-00/compact.done) = No such file or directory: 2.081 sec on http probe failed: 3, retry 2024/11/19-15:18:43.163732 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-02 2024/11/19-15:18:43.163833 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00100/att-02/compact.done) = No such file or directory: 2.713 sec on http probe failed: 4, retry 2024/11/19-15:18:43.199159 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-02 2024/11/19-15:18:43.199234 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00096/att-02/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:18:43.229128 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-02 2024/11/19-15:18:43.229197 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00094/att-02/compact.done) = No such file or directory: 7.184 sec on http probe failed: 3, retry 2024/11/19-15:18:43.541683 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-02 2024/11/19-15:18:43.541751 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00095/att-02/compact.done) = No such file or directory: 7.192 sec on http probe failed: 3, retry 2024/11/19-15:18:43.572252 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-00 2024/11/19-15:18:43.572315 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00103/att-00/compact.done) = No such file or directory: 2.724 sec on http probe failed: 4, retry 2024/11/19-15:18:43.803307 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-02 2024/11/19-15:18:43.803380 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00100/att-02/compact.done) = No such file or directory: 3.355 sec on http probe failed: 5, giveup 2024/11/19-15:18:43.803397 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00100/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.699 sec, real 3.633 (submit 272.743 ms wait 3.355 s), input 31.699 MiB, send shutdown 2024/11/19-15:18:43.812337 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00100/att-02 2024/11/19-15:18:43.812550 139851127064320 [dcompact_etcd.cc:1773] job-00100/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:43.814431 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00100) = success 2024/11/19-15:18:43.814442 139851127064320 [dcompact_etcd.cc:1773] job-00100/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:18:43.814463 139851127064320 (Original Log Time 2024/11/19-15:18:32.844089) EVENT_LOG_v1 {"time_micros": 1732000712844082, "job": 100, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [4567], "score": 0.941722, "input_data_size": 33524498, "oldest_snapshot_seqno": 120983051} 2024/11/19-15:18:43.814471 139851127064320 [compaction_job.cc:2437] [default] [JOB 100] Compacting 1@1 files to L2, score 0.94, subcompactions 1 : 1 2024/11/19-15:18:43.814478 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 98 Base level 1, inputs: [4567(31MB)] 2024/11/19-15:18:43.814733 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004619.sst) /storage/mytopling/..//tmp/Topling-82038-mq26WU 2024/11/19-15:18:43.841376 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004619.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.033 GB 2024/11/19-15:18:43.841490 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004619.sst): index pass time = 0.00's,2851.417'MB/sec index type = UintIndex_AllOne usrkeys = 32051 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0003 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.01 2024/11/19-15:18:43.874026 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004619.sst): first pass time = 0.06's, 560.701'MB/sec 2024/11/19-15:18:43.874071 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004619.sst): old prealloc_size = 36876947, real_size = 32706504 2024/11/19-15:18:43.874092 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004619.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 32051 prefix = 4 raw-key = 0.0003 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.01 raw-val = 0.0326 GB zip-val = 0.0327 GB avg-val =1017.06 avg-zval =1020.19 2024/11/19-15:18:43.900149 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004619.sst): second pass time = 0.00's,92084039.548'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.03's, 1254.628'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.360'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 0.436'MB/sec write SST data time = 0.01's, 4256.794'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.033'GB zip my value throughput = 92084039.548'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 32051 avg-key = 20.00 avg-zkey = 0.01 avg-val = 1017.06 avg-zval = 1020.19 usrkeys = 32051 avg-key = 12.00 avg-zkey = 0.01 avg-val = 1017.06 avg-zval = 1020.19 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0004 GB value = 0.0326 GB dict = 0.00 MB all = 0.0332 GB } __ZipSize{ index = 0.0000 GB value = 0.0327 GB dict = 0.00 MB all = 0.0327 GB } UnZip/Zip{ index =2003.1875 value = 0.9969 dict = 0.00 all = 1.0165 } Zip/UnZip{ index = 0.0005 value = 1.0031 dict = 0.00 all = 0.9838 } ---------------------------- total value len = 4.020501 GB avg = 0.313 KB (by entry num) total key len = 0.326504 GB avg = 0.025 KB total ukey len = 0.223777 GB avg = 0.017 KB total ukey num = 0.012840917 Billion total entry num = 0.012840917 Billion write speed all = 627.094330659 MB/sec (with seq num) write speed all = 612.274994460 MB/sec (without seq num) 2024/11/19-15:18:43.902054 139851127064320 [compaction_job.cc:1948] [default] [JOB 100] Generated table #4619: 32051 keys, 32699466 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:43.902107 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000723902083, "cf_name": "default", "job": 100, "event": "table_file_creation", "file_number": 4619, "file_size": 32699466, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32698128, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641020, "raw_average_key_size": 20, "raw_value_size": 32597750, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32051, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000723, "oldest_key_time": 0, "file_creation_time": 1732000723, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4619, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:43.903021 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-00 2024/11/19-15:18:43.903103 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00104/att-00/compact.done) = No such file or directory: 2.322 sec on http probe failed: 2, retry 2024/11/19-15:18:43.909217 139851127064320 [compaction_job.cc:703] job-00100: subcompact[0], size: 0.032699 G, files: /storage/mytopling/.rocksdb [004619] 2024/11/19-15:18:43.909658 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:18:43.909774 139851127064320 (Original Log Time 2024/11/19-15:18:43.814509) EVENT_LOG_v1 {"time_micros": 1732000723814491, "job": 100, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [4567], "score": 0.941722, "input_data_size": 33524498, "oldest_snapshot_seqno": 120983051} 2024/11/19-15:18:43.909780 139851127064320 (Original Log Time 2024/11/19-15:18:43.909459) [compaction_job.cc:2022] [default] [JOB 100] Compacted 1@1 files to L2 => 32699466 bytes 2024/11/19-15:18:43.909783 139851127064320 (Original Log Time 2024/11/19-15:18:43.909630) [compaction_job.cc:1224] [default] compacted to: files[5 34 32 3 0 0 0] max score 2.42 (53 files need compaction), MB/sec: 353.9 rd, 345.2 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(32.0, 0.0 +0.0 blob) out(31.2 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 32051, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:43.909794 139851127064320 (Original Log Time 2024/11/19-15:18:43.909652) EVENT_LOG_v1 {"time_micros": 1732000723909641, "cf": "default", "job": 100, "event": "compaction_finished", "compaction_time_micros": 94721, "compaction_time_cpu_micros": 94241, "output_level": 2, "num_output_files": 1, "total_output_size": 32699466, "num_input_records": 32051, "num_output_records": 32051, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 34, 32, 3, 0, 0, 0]} 2024/11/19-15:18:43.909873 139851127064320 [compaction_job.cc:2437] [default] [JOB 105] Compacting 1@2 files to L3, score 0.89, subcompactions 1 : 1 2024/11/19-15:18:43.909881 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 103 Base level 2, inputs: [4589(52MB)] 2024/11/19-15:18:44.222377 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-00 2024/11/19-15:18:44.222468 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00103/att-00/compact.done) = No such file or directory: 3.365 sec on http probe failed: 5, giveup 2024/11/19-15:18:44.222485 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00103/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.795 sec, real 3.733 (submit 353.940 ms wait 3.365 s), input 31.795 MiB, send shutdown 2024/11/19-15:18:44.227417 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-00 2024/11/19-15:18:44.227635 139851137550080 [dcompact_etcd.cc:1773] job-00103/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:44.233743 139851127064320 [dcompact_etcd.cc:1163] job-00105/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.323 sec, input = 64.000 MiB 2024/11/19-15:18:44.294210 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-00 2024/11/19-15:18:44.294265 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00102/att-00/compact.done) = No such file or directory: 7.011 sec on http probe failed: 2, retry 2024/11/19-15:18:44.507321 139851137550080 [dcompact_etcd.cc:1163] job-00103/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 31.795 MiB 2024/11/19-15:18:44.940764 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-00 2024/11/19-15:18:44.940845 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00104/att-00/compact.done) = No such file or directory: 3.355 sec on http probe failed: 3, retry 2024/11/19-15:18:45.208409 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-02 2024/11/19-15:18:45.208481 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00096/att-02/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:18:45.236579 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-02 2024/11/19-15:18:45.236633 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00094/att-02/compact.done) = No such file or directory: 9.192 sec on http probe failed: 4, retry 2024/11/19-15:18:45.308741 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-01 2024/11/19-15:18:45.308808 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00103/att-01/compact.done) = No such file or directory: 0.795 sec on http probe failed: 1, retry 2024/11/19-15:18:45.553629 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-02 2024/11/19-15:18:45.553704 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00095/att-02/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:18:45.844082 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-00 2024/11/19-15:18:45.844188 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00105/att-00/compact.done) = No such file or directory: 1.600 sec on http probe failed: 1, retry 2024/11/19-15:18:45.952770 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-01 2024/11/19-15:18:45.952842 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00103/att-01/compact.done) = No such file or directory: 1.437 sec on http probe failed: 2, retry 2024/11/19-15:18:45.977079 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-00 2024/11/19-15:18:45.977137 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00104/att-00/compact.done) = No such file or directory: 4.393 sec on http probe failed: 4, retry 2024/11/19-15:18:46.300151 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-00 2024/11/19-15:18:46.300222 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00102/att-00/compact.done) = No such file or directory: 9.020 sec on http probe failed: 3, retry 2024/11/19-15:18:46.597393 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-01 2024/11/19-15:18:46.597455 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00103/att-01/compact.done) = No such file or directory: 2.082 sec on http probe failed: 3, retry 2024/11/19-15:18:47.013114 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-00 2024/11/19-15:18:47.013213 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00104/att-00/compact.done) = No such file or directory: 5.429 sec on http probe failed: 5, giveup 2024/11/19-15:18:47.013238 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00104/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 51.378 sec, real 5.733 (submit 295.508 ms wait 5.429 s), input 51.378 MiB, send shutdown 2024/11/19-15:18:47.019276 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-00 2024/11/19-15:18:47.019587 139851158521600 [dcompact_etcd.cc:1773] job-00104/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:47.129395 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-00 2024/11/19-15:18:47.129476 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00105/att-00/compact.done) = No such file or directory: 2.891 sec on http probe failed: 2, retry 2024/11/19-15:18:47.216256 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-02 2024/11/19-15:18:47.216332 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00096/att-02/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:18:47.241495 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-01 2024/11/19-15:18:47.241555 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00103/att-01/compact.done) = No such file or directory: 2.726 sec on http probe failed: 4, retry 2024/11/19-15:18:47.242230 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-02 2024/11/19-15:18:47.242277 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00094/att-02/compact.done) = No such file or directory: 11.199 sec on http probe failed: 5, giveup 2024/11/19-15:18:47.242292 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00094/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.812 sec, real 11.481 (submit 276.537 ms wait 11.199 s), input 126.812 MiB, send shutdown 2024/11/19-15:18:47.249033 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00094/att-02 2024/11/19-15:18:47.249242 139851179493120 [dcompact_etcd.cc:1773] job-00094/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:47.251111 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00094) = success 2024/11/19-15:18:47.251119 139851179493120 [dcompact_etcd.cc:1773] job-00094/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:18:47.251139 139851179493120 (Original Log Time 2024/11/19-15:18:12.761423) EVENT_LOG_v1 {"time_micros": 1732000692761416, "job": 94, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4536, 4537, 4538, 4539], "score": 2.26731, "input_data_size": 134099016, "oldest_snapshot_seqno": 119851572} 2024/11/19-15:18:47.251146 139851179493120 [compaction_job.cc:2437] [default] [JOB 94] Compacting 4@1 files to L2, score 2.27, subcompactions 1 : 1 2024/11/19-15:18:47.251154 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 92 Base level 1, inputs: [4536(31MB) 4537(31MB) 4538(31MB) 4539(31MB)] 2024/11/19-15:18:47.251416 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004620.sst) /storage/mytopling/..//tmp/Topling-82038-p0Z124 2024/11/19-15:18:47.303007 139851158521600 [dcompact_etcd.cc:1163] job-00104/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.283 sec, input = 51.378 MiB 2024/11/19-15:18:47.304774 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004620.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:18:47.310124 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004620.sst): index pass time = 0.01's, 97.293'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64716 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:18:47.372467 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004620.sst): first pass time = 0.12's, 554.461'MB/sec 2024/11/19-15:18:47.372525 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004620.sst): old prealloc_size = 73819750, real_size = 66152480 2024/11/19-15:18:47.372583 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004620.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64716 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.99 avg-zval =1020.24 2024/11/19-15:18:47.372652 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004620.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.068 sec, Key+Value = 0.067 GB 2024/11/19-15:18:47.432223 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004620.sst): second pass time = 0.00's,140331296.375'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1108.187'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2040.133'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.378'MB/sec write SST data time = 0.02's, 3926.489'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 140331296.375'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64716 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 usrkeys = 64716 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0466 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 4.086316 GB avg = 0.317 KB (by entry num) total key len = 0.327799 GB avg = 0.025 KB total ukey len = 0.224554 GB avg = 0.017 KB total ukey num = 0.012905633 Billion total entry num = 0.012905633 Billion write speed all = 421.836168610 MB/sec (with seq num) write speed all = 411.969523940 MB/sec (without seq num) 2024/11/19-15:18:47.435720 139851179493120 [compaction_job.cc:1948] [default] [JOB 94] Generated table #4620: 64716 keys, 66137732 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:47.435781 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000727435756, "cf_name": "default", "job": 94, "event": "table_file_creation", "file_number": 4620, "file_size": 66137732, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025936, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294320, "raw_average_key_size": 20, "raw_value_size": 65815378, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64716, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000727, "oldest_key_time": 0, "file_creation_time": 1732000727, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4620, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:47.449926 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004621.sst) /storage/mytopling/..//tmp/Topling-82038-UK44ow 2024/11/19-15:18:47.503555 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004621.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:18:47.503683 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004621.sst): index pass time = 0.00's,5146.532'MB/sec index type = UintIndex_AllOne usrkeys = 63514 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:18:47.563188 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-02 2024/11/19-15:18:47.563270 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00095/att-02/compact.done) = No such file or directory: 11.209 sec on http probe failed: 5, giveup 2024/11/19-15:18:47.563288 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00095/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.833 sec, real 11.488 (submit 269.799 ms wait 11.209 s), input 126.833 MiB, send shutdown 2024/11/19-15:18:47.568227 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00095/att-02 2024/11/19-15:18:47.568440 139850978166528 [dcompact_etcd.cc:1773] job-00095/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:47.569157 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004621.sst): first pass time = 0.12's, 552.464'MB/sec 2024/11/19-15:18:47.569200 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004621.sst): old prealloc_size = 73819750, real_size = 64814856 2024/11/19-15:18:47.569222 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004621.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63514 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0646 GB zip-val = 0.0648 GB avg-val =1016.97 avg-zval =1020.23 2024/11/19-15:18:47.570862 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00095) = success 2024/11/19-15:18:47.570873 139850978166528 [dcompact_etcd.cc:1773] job-00095/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:18:47.570894 139850978166528 (Original Log Time 2024/11/19-15:18:13.056496) EVENT_LOG_v1 {"time_micros": 1732000693056489, "job": 95, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4540, 4541, 4542, 4543], "score": 2.01754, "input_data_size": 134100360, "oldest_snapshot_seqno": 119864574} 2024/11/19-15:18:47.570901 139850978166528 [compaction_job.cc:2437] [default] [JOB 95] Compacting 4@1 files to L2, score 2.02, subcompactions 1 : 1 2024/11/19-15:18:47.570909 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 93 Base level 1, inputs: [4540(31MB) 4541(31MB) 4542(31MB) 4543(31MB)] 2024/11/19-15:18:47.571192 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004622.sst) /storage/mytopling/..//tmp/Topling-82038-ZyyO8B 2024/11/19-15:18:47.623610 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004621.sst): second pass time = 0.00's,241014861.940'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1191.345'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.150'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.414'MB/sec write SST data time = 0.02's, 4050.390'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 241014861.940'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63514 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.97 avg-zval = 1020.23 usrkeys = 63514 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.97 avg-zval = 1020.23 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0000 GB value = 0.0648 GB dict = 0.00 MB all = 0.0648 GB } UnZip/Zip{ index =3969.6250 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 4.150908 GB avg = 0.320 KB (by entry num) total key len = 0.329069 GB avg = 0.025 KB total ukey len = 0.225316 GB avg = 0.017 KB total ukey num = 0.012969147 Billion total entry num = 0.012969147 Billion write speed all = 420.440436312 MB/sec (with seq num) write speed all = 410.703326052 MB/sec (without seq num) 2024/11/19-15:18:47.627129 139851179493120 [compaction_job.cc:1948] [default] [JOB 94] Generated table #4621: 63514 keys, 64799946 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:47.627183 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000727627159, "cf_name": "default", "job": 94, "event": "table_file_creation", "file_number": 4621, "file_size": 64799946, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64798608, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1270280, "raw_average_key_size": 20, "raw_value_size": 64591983, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63514, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000727, "oldest_key_time": 0, "file_creation_time": 1732000727, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4621, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:47.627428 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004622.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:18:47.630757 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004622.sst): index pass time = 0.00's, 156.723'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64716 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:18:47.641608 139851179493120 [compaction_job.cc:703] job-00094: subcompact[0], size: 0.130938 G, files: /storage/mytopling/.rocksdb [004620,004621] 2024/11/19-15:18:47.642235 139851179493120 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:18:47.642362 139851179493120 (Original Log Time 2024/11/19-15:18:47.251184) EVENT_LOG_v1 {"time_micros": 1732000727251166, "job": 94, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4536, 4537, 4538, 4539], "score": 2.26731, "input_data_size": 134099016, "oldest_snapshot_seqno": 119851572} 2024/11/19-15:18:47.642369 139851179493120 (Original Log Time 2024/11/19-15:18:47.642027) [compaction_job.cc:2022] [default] [JOB 94] Compacted 4@1 files to L2 => 130937678 bytes 2024/11/19-15:18:47.642372 139851179493120 (Original Log Time 2024/11/19-15:18:47.642204) [compaction_job.cc:1224] [default] compacted to: files[5 30 34 3 0 0 0] max score 2.75 (54 files need compaction), MB/sec: 343.5 rd, 335.4 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(124.9 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128230, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:47.642387 139851179493120 (Original Log Time 2024/11/19-15:18:47.642228) EVENT_LOG_v1 {"time_micros": 1732000727642216, "cf": "default", "job": 94, "event": "compaction_finished", "compaction_time_micros": 390436, "compaction_time_cpu_micros": 388666, "output_level": 2, "num_output_files": 2, "total_output_size": 130937678, "num_input_records": 128230, "num_output_records": 128230, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 30, 34, 3, 0, 0, 0]} 2024/11/19-15:18:47.647234 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004510.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:47.647259 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000727647256, "job": 94, "event": "table_file_deletion", "file_number": 4510} 2024/11/19-15:18:47.651826 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004511.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:47.651847 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000727651845, "job": 94, "event": "table_file_deletion", "file_number": 4511} 2024/11/19-15:18:47.656743 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004512.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:47.656763 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000727656761, "job": 94, "event": "table_file_deletion", "file_number": 4512} 2024/11/19-15:18:47.661486 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004513.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:47.661506 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000727661504, "job": 94, "event": "table_file_deletion", "file_number": 4513} 2024/11/19-15:18:47.661630 139851179493120 [compaction_job.cc:2437] [default] [JOB 106] Compacting 1@1 files to L2, score 0.93, subcompactions 1 : 1 2024/11/19-15:18:47.661638 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 104 Base level 1, inputs: [4568(31MB)] 2024/11/19-15:18:47.694100 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004622.sst): first pass time = 0.12's, 546.066'MB/sec 2024/11/19-15:18:47.694145 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004622.sst): old prealloc_size = 73819750, real_size = 66152560 2024/11/19-15:18:47.694199 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004622.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64716 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.99 avg-zval =1020.24 2024/11/19-15:18:47.694257 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004622.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.067 GB 2024/11/19-15:18:47.753758 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004622.sst): second pass time = 0.00's,203763049.536'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1109.881'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2194.635'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.377'MB/sec write SST data time = 0.02's, 3972.611'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 203763049.536'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64716 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 usrkeys = 64716 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0466 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 4.216724 GB avg = 0.324 KB (by entry num) total key len = 0.330363 GB avg = 0.025 KB total ukey len = 0.226092 GB avg = 0.017 KB total ukey num = 0.013033863 Billion total entry num = 0.013033863 Billion write speed all = 421.589437897 MB/sec (with seq num) write speed all = 411.921816997 MB/sec (without seq num) 2024/11/19-15:18:47.757382 139850978166528 [compaction_job.cc:1948] [default] [JOB 95] Generated table #4622: 64716 keys, 66137796 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:47.757433 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000727757409, "cf_name": "default", "job": 95, "event": "table_file_creation", "file_number": 4622, "file_size": 66137796, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66026016, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294320, "raw_average_key_size": 20, "raw_value_size": 65815465, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64716, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000727, "oldest_key_time": 0, "file_creation_time": 1732000727, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4622, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:47.772073 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004623.sst) /storage/mytopling/..//tmp/Topling-82038-nCIGjq 2024/11/19-15:18:47.824280 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004623.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:18:47.829421 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004623.sst): index pass time = 0.01's, 99.554'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63537 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:18:47.885719 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-01 2024/11/19-15:18:47.885807 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00103/att-01/compact.done) = No such file or directory: 3.370 sec on http probe failed: 5, giveup 2024/11/19-15:18:47.885825 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00103/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.795 sec, real 3.658 (submit 279.590 ms wait 3.370 s), input 31.795 MiB, send shutdown 2024/11/19-15:18:47.893872 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-01 2024/11/19-15:18:47.894115 139851137550080 [dcompact_etcd.cc:1773] job-00103/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:47.899336 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004623.sst): first pass time = 0.13's, 517.762'MB/sec 2024/11/19-15:18:47.899379 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004623.sst): old prealloc_size = 73819750, real_size = 64945104 2024/11/19-15:18:47.899433 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004623.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63537 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0646 GB zip-val = 0.0648 GB avg-val =1016.95 avg-zval =1020.20 2024/11/19-15:18:47.899493 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004623.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.075 sec, Key+Value = 0.066 GB 2024/11/19-15:18:47.957709 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004623.sst): second pass time = 0.00's,213953721.854'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1113.474'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2196.155'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.385'MB/sec write SST data time = 0.02's, 3816.744'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 213953721.854'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63537 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.95 avg-zval = 1020.20 usrkeys = 63537 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.95 avg-zval = 1020.20 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7916 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0648 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index = 7.0409 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 4.281338 GB avg = 0.327 KB (by entry num) total key len = 0.331634 GB avg = 0.025 KB total ukey len = 0.226855 GB avg = 0.017 KB total ukey num = 0.013097400 Billion total entry num = 0.013097400 Billion write speed all = 419.760318657 MB/sec (with seq num) write speed all = 410.225867956 MB/sec (without seq num) 2024/11/19-15:18:47.961498 139850978166528 [compaction_job.cc:1948] [default] [JOB 95] Generated table #4623: 63537 keys, 64930564 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:47.961550 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000727961526, "cf_name": "default", "job": 95, "event": "table_file_creation", "file_number": 4623, "file_size": 64930564, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64820736, "index_size": 108288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1270740, "raw_average_key_size": 20, "raw_value_size": 64614024, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63537, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000727, "oldest_key_time": 0, "file_creation_time": 1732000727, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4623, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:47.966562 139851179493120 [dcompact_etcd.cc:1163] job-00106/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.304 sec, input = 31.701 MiB 2024/11/19-15:18:47.975857 139850978166528 [compaction_job.cc:703] job-00095: subcompact[0], size: 0.131068 G, files: /storage/mytopling/.rocksdb [004622,004623] 2024/11/19-15:18:47.976513 139850978166528 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:18:47.976642 139850978166528 (Original Log Time 2024/11/19-15:18:47.570949) EVENT_LOG_v1 {"time_micros": 1732000727570931, "job": 95, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4540, 4541, 4542, 4543], "score": 2.01754, "input_data_size": 134100360, "oldest_snapshot_seqno": 119864574} 2024/11/19-15:18:47.976648 139850978166528 (Original Log Time 2024/11/19-15:18:47.976312) [compaction_job.cc:2022] [default] [JOB 95] Compacted 4@1 files to L2 => 131068360 bytes 2024/11/19-15:18:47.976651 139850978166528 (Original Log Time 2024/11/19-15:18:47.976482) [compaction_job.cc:1224] [default] compacted to: files[5 26 36 3 0 0 0] max score 3.19 (55 files need compaction), MB/sec: 331.2 rd, 323.7 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128253, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:47.976666 139850978166528 (Original Log Time 2024/11/19-15:18:47.976506) EVENT_LOG_v1 {"time_micros": 1732000727976494, "cf": "default", "job": 95, "event": "compaction_finished", "compaction_time_micros": 404920, "compaction_time_cpu_micros": 400959, "output_level": 2, "num_output_files": 2, "total_output_size": 131068360, "num_input_records": 128253, "num_output_records": 128253, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 26, 36, 3, 0, 0, 0]} 2024/11/19-15:18:47.981143 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004514.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:47.981166 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000727981163, "job": 95, "event": "table_file_deletion", "file_number": 4514} 2024/11/19-15:18:47.982035 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004515.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:47.982052 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000727982050, "job": 95, "event": "table_file_deletion", "file_number": 4515} 2024/11/19-15:18:47.983327 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004516.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:47.983345 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000727983343, "job": 95, "event": "table_file_deletion", "file_number": 4516} 2024/11/19-15:18:47.987642 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004517.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:47.987660 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000727987658, "job": 95, "event": "table_file_deletion", "file_number": 4517} 2024/11/19-15:18:47.987776 139850978166528 [compaction_job.cc:2437] [default] [JOB 107] Compacting 4@2 files to L3, score 1.01, subcompactions 1 : 1 2024/11/19-15:18:47.987785 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 105 Base level 2, inputs: [4581(26MB) 4583(240KB) 4584(10MB) 4591(52MB)] 2024/11/19-15:18:48.199159 139851137550080 [dcompact_etcd.cc:1163] job-00103/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.305 sec, input = 31.795 MiB 2024/11/19-15:18:48.297801 139850978166528 [dcompact_etcd.cc:1163] job-00107/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.310 sec, input = 121.222 MiB 2024/11/19-15:18:48.308157 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-00 2024/11/19-15:18:48.308221 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00102/att-00/compact.done) = No such file or directory: 11.026 sec on http probe failed: 4, retry 2024/11/19-15:18:48.418194 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-00 2024/11/19-15:18:48.418250 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00105/att-00/compact.done) = No such file or directory: 4.176 sec on http probe failed: 3, retry 2024/11/19-15:18:48.596259 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-01 2024/11/19-15:18:48.596327 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00104/att-01/compact.done) = No such file or directory: 1.285 sec on http probe failed: 1, retry 2024/11/19-15:18:48.767755 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-00 2024/11/19-15:18:48.767835 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00106/att-00/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:18:49.000465 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-02 2024/11/19-15:18:49.000535 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00103/att-02/compact.done) = No such file or directory: 0.795 sec on http probe failed: 1, retry 2024/11/19-15:18:49.223299 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-02 2024/11/19-15:18:49.223372 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00096/att-02/compact.done) = No such file or directory: 11.205 sec on http probe failed: 5, giveup 2024/11/19-15:18:49.223390 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00096/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.961 sec, real 11.475 (submit 262.983 ms wait 11.205 s), input 126.961 MiB, send shutdown 2024/11/19-15:18:49.232244 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00096/att-02 2024/11/19-15:18:49.232457 139851169007360 [dcompact_etcd.cc:1773] job-00096/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:49.236024 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00096) = success 2024/11/19-15:18:49.236032 139851169007360 [dcompact_etcd.cc:1773] job-00096/att-03: CleanFiles 0.004 sec : 0.000 2024/11/19-15:18:49.236053 139851169007360 (Original Log Time 2024/11/19-15:18:14.720201) EVENT_LOG_v1 {"time_micros": 1732000694720194, "job": 96, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4544, 4545, 4546, 4547], "score": 1.76775, "input_data_size": 134100552, "oldest_snapshot_seqno": 119984343} 2024/11/19-15:18:49.236060 139851169007360 [compaction_job.cc:2437] [default] [JOB 96] Compacting 4@1 files to L2, score 1.77, subcompactions 1 : 1 2024/11/19-15:18:49.236067 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 94 Base level 1, inputs: [4544(31MB) 4545(31MB) 4546(31MB) 4547(31MB)] 2024/11/19-15:18:49.236312 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004624.sst) /storage/mytopling/..//tmp/Topling-82038-0ahPxG 2024/11/19-15:18:49.291831 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004624.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:18:49.291958 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004624.sst): index pass time = 0.00's,5256.277'MB/sec index type = UintIndex_AllOne usrkeys = 64712 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:18:49.361567 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004624.sst): first pass time = 0.13's, 535.838'MB/sec 2024/11/19-15:18:49.361620 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004624.sst): old prealloc_size = 73819750, real_size = 66042008 2024/11/19-15:18:49.361644 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004624.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64712 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.04 avg-zval =1020.30 2024/11/19-15:18:49.410966 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-00 2024/11/19-15:18:49.411062 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00106/att-00/compact.done) = No such file or directory: 1.435 sec on http probe failed: 2, retry 2024/11/19-15:18:49.419341 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004624.sst): second pass time = 0.00's,173197265.789'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1144.129'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.136'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.404'MB/sec write SST data time = 0.02's, 3741.312'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 173197265.789'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64712 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.04 avg-zval = 1020.30 usrkeys = 64712 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.04 avg-zval = 1020.30 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.5000 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 4.347153 GB avg = 0.330 KB (by entry num) total key len = 0.332928 GB avg = 0.025 KB total ukey len = 0.227631 GB avg = 0.017 KB total ukey num = 0.013162112 Billion total entry num = 0.013162112 Billion write speed all = 375.874651768 MB/sec (with seq num) write speed all = 367.417867423 MB/sec (without seq num) 2024/11/19-15:18:49.436781 139851169007360 [compaction_job.cc:1948] [default] [JOB 96] Generated table #4624: 64712 keys, 66026826 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:49.436833 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729436810, "cf_name": "default", "job": 96, "event": "table_file_creation", "file_number": 4624, "file_size": 66026826, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025504, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294240, "raw_average_key_size": 20, "raw_value_size": 65814961, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64712, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000729, "oldest_key_time": 0, "file_creation_time": 1732000729, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4624, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:49.453403 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004625.sst) /storage/mytopling/..//tmp/Topling-82038-OwVFxC 2024/11/19-15:18:49.510415 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004625.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:18:49.513712 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004625.sst): index pass time = 0.00's, 156.122'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63664 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:18:49.578728 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004625.sst): first pass time = 0.13's, 526.868'MB/sec 2024/11/19-15:18:49.578780 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004625.sst): old prealloc_size = 73819750, real_size = 65077664 2024/11/19-15:18:49.578836 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004625.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63664 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0650 GB avg-val =1017.00 avg-zval =1020.25 2024/11/19-15:18:49.578878 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004625.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.068 sec, Key+Value = 0.066 GB 2024/11/19-15:18:49.629415 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-01 2024/11/19-15:18:49.629500 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00104/att-01/compact.done) = No such file or directory: 2.321 sec on http probe failed: 2, retry 2024/11/19-15:18:49.638507 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004625.sst): second pass time = 0.00's,160660158.809'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1089.542'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2052.906'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.374'MB/sec write SST data time = 0.02's, 3810.340'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 160660158.809'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63664 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.25 usrkeys = 63664 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7933 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0650 GB dict = 0.00 MB all = 0.0651 GB } UnZip/Zip{ index = 7.0466 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 4.411899 GB avg = 0.334 KB (by entry num) total key len = 0.334202 GB avg = 0.025 KB total ukey len = 0.228395 GB avg = 0.017 KB total ukey num = 0.013225776 Billion total entry num = 0.013225776 Billion write speed all = 374.583490543 MB/sec (with seq num) write speed all = 366.232790473 MB/sec (without seq num) 2024/11/19-15:18:49.642677 139851169007360 [compaction_job.cc:1948] [default] [JOB 96] Generated table #4625: 63664 keys, 65063172 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:49.642740 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729642716, "cf_name": "default", "job": 96, "event": "table_file_creation", "file_number": 4625, "file_size": 65063172, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64953168, "index_size": 108416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1273280, "raw_average_key_size": 20, "raw_value_size": 64746044, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63664, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000729, "oldest_key_time": 0, "file_creation_time": 1732000729, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4625, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:49.644020 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-02 2024/11/19-15:18:49.644076 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00103/att-02/compact.done) = No such file or directory: 1.437 sec on http probe failed: 2, retry 2024/11/19-15:18:49.657964 139851169007360 [compaction_job.cc:703] job-00096: subcompact[0], size: 0.131090 G, files: /storage/mytopling/.rocksdb [004624,004625] 2024/11/19-15:18:49.658531 139851169007360 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:18:49.658678 139851169007360 (Original Log Time 2024/11/19-15:18:49.236097) EVENT_LOG_v1 {"time_micros": 1732000729236080, "job": 96, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4544, 4545, 4546, 4547], "score": 1.76775, "input_data_size": 134100552, "oldest_snapshot_seqno": 119984343} 2024/11/19-15:18:49.658684 139851169007360 (Original Log Time 2024/11/19-15:18:49.658338) [compaction_job.cc:2022] [default] [JOB 96] Compacted 4@1 files to L2 => 131089998 bytes 2024/11/19-15:18:49.658687 139851169007360 (Original Log Time 2024/11/19-15:18:49.658502) [compaction_job.cc:1224] [default] compacted to: files[5 22 38 3 0 0 0] max score 3.78 (53 files need compaction), MB/sec: 317.9 rd, 310.7 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128376, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:49.658703 139851169007360 (Original Log Time 2024/11/19-15:18:49.658524) EVENT_LOG_v1 {"time_micros": 1732000729658513, "cf": "default", "job": 96, "event": "compaction_finished", "compaction_time_micros": 421878, "compaction_time_cpu_micros": 406496, "output_level": 2, "num_output_files": 2, "total_output_size": 131089998, "num_input_records": 128376, "num_output_records": 128376, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 22, 38, 3, 0, 0, 0]} 2024/11/19-15:18:49.663039 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004518.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.663061 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729663059, "job": 96, "event": "table_file_deletion", "file_number": 4518} 2024/11/19-15:18:49.667278 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004519.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.667297 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729667295, "job": 96, "event": "table_file_deletion", "file_number": 4519} 2024/11/19-15:18:49.671417 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004520.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.671436 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729671434, "job": 96, "event": "table_file_deletion", "file_number": 4520} 2024/11/19-15:18:49.675458 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004521.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.675476 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729675474, "job": 96, "event": "table_file_deletion", "file_number": 4521} 2024/11/19-15:18:49.679637 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004522.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.679655 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729679654, "job": 96, "event": "table_file_deletion", "file_number": 4522} 2024/11/19-15:18:49.684022 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004523.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.684040 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729684039, "job": 96, "event": "table_file_deletion", "file_number": 4523} 2024/11/19-15:18:49.688524 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004524.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.688543 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729688541, "job": 96, "event": "table_file_deletion", "file_number": 4524} 2024/11/19-15:18:49.692887 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004525.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.692905 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729692903, "job": 96, "event": "table_file_deletion", "file_number": 4525} 2024/11/19-15:18:49.697173 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004526.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.697192 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729697190, "job": 96, "event": "table_file_deletion", "file_number": 4526} 2024/11/19-15:18:49.701293 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004527.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.701311 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729701310, "job": 96, "event": "table_file_deletion", "file_number": 4527} 2024/11/19-15:18:49.702918 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-00 2024/11/19-15:18:49.702983 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00105/att-00/compact.done) = No such file or directory: 5.465 sec on http probe failed: 4, retry 2024/11/19-15:18:49.705561 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004528.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.705580 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729705578, "job": 96, "event": "table_file_deletion", "file_number": 4528} 2024/11/19-15:18:49.709647 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004529.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.709665 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729709664, "job": 96, "event": "table_file_deletion", "file_number": 4529} 2024/11/19-15:18:49.709822 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004530.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.709832 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729709831, "job": 96, "event": "table_file_deletion", "file_number": 4530} 2024/11/19-15:18:49.709984 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004531.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.709995 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729709994, "job": 96, "event": "table_file_deletion", "file_number": 4531} 2024/11/19-15:18:49.714098 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004532.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.714117 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729714115, "job": 96, "event": "table_file_deletion", "file_number": 4532} 2024/11/19-15:18:49.718375 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004533.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.718393 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729718391, "job": 96, "event": "table_file_deletion", "file_number": 4533} 2024/11/19-15:18:49.722491 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004534.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.722510 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729722508, "job": 96, "event": "table_file_deletion", "file_number": 4534} 2024/11/19-15:18:49.726809 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004535.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.726827 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729726825, "job": 96, "event": "table_file_deletion", "file_number": 4535} 2024/11/19-15:18:49.726953 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004575.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.726962 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729726961, "job": 96, "event": "table_file_deletion", "file_number": 4575} 2024/11/19-15:18:49.727085 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004576.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:18:49.727093 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000729727092, "job": 96, "event": "table_file_deletion", "file_number": 4576} 2024/11/19-15:18:49.727209 139851169007360 [compaction_job.cc:2437] [default] [JOB 108] Compacting 1@2 files to L3, score 1.01, subcompactions 1 : 1 2024/11/19-15:18:49.727217 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 106 Base level 2, inputs: [4592(52MB)] 2024/11/19-15:18:50.017798 139851169007360 [dcompact_etcd.cc:1163] job-00108/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.290 sec, input = 63.818 MiB 2024/11/19-15:18:50.053055 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-00 2024/11/19-15:18:50.053114 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00106/att-00/compact.done) = No such file or directory: 2.079 sec on http probe failed: 3, retry 2024/11/19-15:18:50.288835 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-02 2024/11/19-15:18:50.288899 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00103/att-02/compact.done) = No such file or directory: 2.081 sec on http probe failed: 3, retry 2024/11/19-15:18:50.317132 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-00 2024/11/19-15:18:50.317191 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00102/att-00/compact.done) = No such file or directory: 13.034 sec on http probe failed: 5, giveup 2024/11/19-15:18:50.317209 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00102/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 254.856 sec, real 13.303 (submit 260.820 ms wait 13.034 s), input 254.856 MiB, send shutdown 2024/11/19-15:18:50.321918 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-00 2024/11/19-15:18:50.322194 139851148035840 [dcompact_etcd.cc:1773] job-00102/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:50.363754 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4626. Immutable memtables: 0. 2024/11/19-15:18:50.364245 139852362282752 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16273899 2024/11/19-15:18:50.364337 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 109] Syncing log #4604 2024/11/19-15:18:50.364431 139850967680768 (Original Log Time 2024/11/19-15:18:50.364324) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:18:50.364439 139850967680768 [flush_job.cc:879] [default] [JOB 109] Flushing memtable with next log file: 4626 2024/11/19-15:18:50.364458 139850967680768 EVENT_LOG_v1 {"time_micros": 1732000730364453, "job": 109, "event": "flush_started", "num_memtables": 1, "num_entries": 1143706, "num_deletes": 0, "total_data_size": 464989395, "memory_usage": 536870280, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:18:50.364462 139850967680768 [flush_job.cc:910] [default] [JOB 109] Level-0 flush table #4627: started 2024/11/19-15:18:50.364764 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000011.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004627.sst in ConvertToSST, mem_size = 511.999 M, time = 293.271 us 2024/11/19-15:18:50.538911 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004627.sst): fsize = 512.001 M, time(ms): open: 0.029, seek: 0.014, finish: 0.034, meta: 0.012, Flush: 0.013, sync: 174.021, close: 0.009, all: 174.132 2024/11/19-15:18:50.540125 139850967680768 [flush_job.cc:1083] [default] [JOB 109] Flush lasted 175694 microseconds, and 70784 cpu microseconds. 2024/11/19-15:18:50.561082 139850967680768 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 15785682 2024/11/19-15:18:50.561170 139850967680768 (Original Log Time 2024/11/19-15:18:50.538961) [flush_job.cc:1029] [default] [JOB 109] Level-0 flush table #4627: 536871706 bytes OK (needs compaction) 2024/11/19-15:18:50.561178 139850967680768 (Original Log Time 2024/11/19-15:18:50.540142) [memtable_list.cc:560] [default] Level-0 commit table #4627 started 2024/11/19-15:18:50.561182 139850967680768 (Original Log Time 2024/11/19-15:18:50.561050) [memtable_list.cc:759] [default] Level-0 commit table #4627: memtable #1 done 2024/11/19-15:18:50.561186 139850967680768 (Original Log Time 2024/11/19-15:18:50.561075) EVENT_LOG_v1 {"time_micros": 1732000730561068, "job": 109, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [6, 22, 38, 3, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:18:50.561190 139850967680768 (Original Log Time 2024/11/19-15:18:50.561138) [db_impl_compaction_flush.cc:310] [default] Level summary: files[6 22 38 3 0 0 0] max score 4.53 (53 files need compaction) 2024/11/19-15:18:50.624008 139851148035840 [dcompact_etcd.cc:1163] job-00102/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.302 sec, input = 254.856 MiB 2024/11/19-15:18:50.666177 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-01 2024/11/19-15:18:50.666257 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00104/att-01/compact.done) = No such file or directory: 3.354 sec on http probe failed: 3, retry 2024/11/19-15:18:50.692886 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-00 2024/11/19-15:18:50.692982 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00106/att-00/compact.done) = No such file or directory: 2.721 sec on http probe failed: 4, retry 2024/11/19-15:18:50.933884 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-02 2024/11/19-15:18:50.933995 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00103/att-02/compact.done) = No such file or directory: 2.726 sec on http probe failed: 4, retry 2024/11/19-15:18:50.988062 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-00 2024/11/19-15:18:50.988140 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00105/att-00/compact.done) = No such file or directory: 6.749 sec on http probe failed: 5, giveup 2024/11/19-15:18:50.988158 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00105/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 64.000 sec, real 7.078 (submit 323.389 ms wait 6.749 s), input 64.000 MiB, send shutdown 2024/11/19-15:18:50.992902 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-00 2024/11/19-15:18:50.993140 139851127064320 [dcompact_etcd.cc:1773] job-00105/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:51.287249 139851127064320 [dcompact_etcd.cc:1163] job-00105/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.294 sec, input = 64.000 MiB 2024/11/19-15:18:51.335832 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-00 2024/11/19-15:18:51.335898 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00106/att-00/compact.done) = No such file or directory: 3.361 sec on http probe failed: 5, giveup 2024/11/19-15:18:51.335914 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00106/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.701 sec, real 3.674 (submit 304.462 ms wait 3.361 s), input 31.701 MiB, send shutdown 2024/11/19-15:18:51.336674 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-00 2024/11/19-15:18:51.336735 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00107/att-00/compact.done) = No such file or directory: 3.031 sec on http probe failed: 1, retry 2024/11/19-15:18:51.343727 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-00 2024/11/19-15:18:51.343945 139851179493120 [dcompact_etcd.cc:1773] job-00106/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:51.574745 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-02 2024/11/19-15:18:51.574811 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00103/att-02/compact.done) = No such file or directory: 3.371 sec on http probe failed: 5, giveup 2024/11/19-15:18:51.574827 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00103/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.795 sec, real 3.681 (submit 304.922 ms wait 3.371 s), input 31.795 MiB, send shutdown 2024/11/19-15:18:51.583031 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00103/att-02 2024/11/19-15:18:51.583232 139851137550080 [dcompact_etcd.cc:1773] job-00103/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:51.584996 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00103) = success 2024/11/19-15:18:51.585010 139851137550080 [dcompact_etcd.cc:1773] job-00103/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:18:51.585032 139851137550080 (Original Log Time 2024/11/19-15:18:40.488734) EVENT_LOG_v1 {"time_micros": 1732000720488720, "job": 103, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [4554], "score": 0.879029, "input_data_size": 33524882, "oldest_snapshot_seqno": 121620765} 2024/11/19-15:18:51.585039 139851137550080 [compaction_job.cc:2437] [default] [JOB 103] Compacting 1@1 files to L2, score 0.88, subcompactions 1 : 1 2024/11/19-15:18:51.585045 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 101 Base level 1, inputs: [4554(31MB)] 2024/11/19-15:18:51.585302 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004628.sst) /storage/mytopling/..//tmp/Topling-82038-QxxYTO 2024/11/19-15:18:51.611149 139851179493120 [dcompact_etcd.cc:1163] job-00106/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.267 sec, input = 31.701 MiB 2024/11/19-15:18:51.612768 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004628.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.033 GB 2024/11/19-15:18:51.612882 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004628.sst): index pass time = 0.00's,2545.685'MB/sec index type = UintIndex_AllOne usrkeys = 32152 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 6 raw-key = 0.0003 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.01 2024/11/19-15:18:51.622592 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-00 2024/11/19-15:18:51.622672 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00108/att-00/compact.done) = No such file or directory: 1.596 sec on http probe failed: 1, retry 2024/11/19-15:18:51.645873 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004628.sst): first pass time = 0.06's, 550.526'MB/sec 2024/11/19-15:18:51.645931 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004628.sst): old prealloc_size = 36877370, real_size = 32805784 2024/11/19-15:18:51.645954 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004628.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 32152 prefix = 4 raw-key = 0.0003 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.01 raw-val = 0.0327 GB zip-val = 0.0328 GB avg-val =1016.94 avg-zval =1020.07 2024/11/19-15:18:51.672708 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004628.sst): second pass time = 0.00's,82567474.747'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.03's, 1225.287'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.186'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 0.422'MB/sec write SST data time = 0.01's, 4246.064'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.033'GB zip my value throughput = 82567474.747'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 32152 avg-key = 20.00 avg-zkey = 0.01 avg-val = 1016.94 avg-zval = 1020.07 usrkeys = 32152 avg-key = 12.00 avg-zkey = 0.01 avg-val = 1016.94 avg-zval = 1020.07 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0004 GB value = 0.0327 GB dict = 0.00 MB all = 0.0333 GB } __ZipSize{ index = 0.0000 GB value = 0.0328 GB dict = 0.00 MB all = 0.0328 GB } UnZip/Zip{ index =2009.5000 value = 0.9969 dict = 0.00 all = 1.0165 } Zip/UnZip{ index = 0.0005 value = 1.0031 dict = 0.00 all = 0.9838 } ---------------------------- total value len = 4.444595 GB avg = 0.335 KB (by entry num) total key len = 0.334845 GB avg = 0.025 KB total ukey len = 0.228781 GB avg = 0.017 KB total ukey num = 0.013257928 Billion total entry num = 0.013257928 Billion write speed all = 325.031680147 MB/sec (with seq num) write speed all = 317.818706728 MB/sec (without seq num) 2024/11/19-15:18:51.674636 139851137550080 [compaction_job.cc:1948] [default] [JOB 103] Generated table #4628: 32152 keys, 32798730 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:51.674695 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000731674671, "cf_name": "default", "job": 103, "event": "table_file_creation", "file_number": 4628, "file_size": 32798730, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32797408, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643040, "raw_average_key_size": 20, "raw_value_size": 32696720, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32152, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000731, "oldest_key_time": 0, "file_creation_time": 1732000731, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4628, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:51.682451 139851137550080 [compaction_job.cc:703] job-00103: subcompact[0], size: 0.032799 G, files: /storage/mytopling/.rocksdb [004628] 2024/11/19-15:18:51.682910 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16273898 2024/11/19-15:18:51.683051 139851137550080 (Original Log Time 2024/11/19-15:18:51.585080) EVENT_LOG_v1 {"time_micros": 1732000731585057, "job": 103, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [4554], "score": 0.879029, "input_data_size": 33524882, "oldest_snapshot_seqno": 121620765} 2024/11/19-15:18:51.683057 139851137550080 (Original Log Time 2024/11/19-15:18:51.682716) [compaction_job.cc:2022] [default] [JOB 103] Compacted 1@1 files to L2 => 32798730 bytes 2024/11/19-15:18:51.683061 139851137550080 (Original Log Time 2024/11/19-15:18:51.682881) [compaction_job.cc:1224] [default] compacted to: files[6 21 39 3 0 0 0] max score 4.76 (54 files need compaction), MB/sec: 344.3 rd, 336.8 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(32.0, 0.0 +0.0 blob) out(31.3 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 32152, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:51.683076 139851137550080 (Original Log Time 2024/11/19-15:18:51.682903) EVENT_LOG_v1 {"time_micros": 1732000731682892, "cf": "default", "job": 103, "event": "compaction_finished", "compaction_time_micros": 97383, "compaction_time_cpu_micros": 96933, "output_level": 2, "num_output_files": 1, "total_output_size": 32798730, "num_input_records": 32152, "num_output_records": 32152, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 21, 39, 3, 0, 0, 0]} 2024/11/19-15:18:51.683139 139851137550080 [compaction_job.cc:2437] [default] [JOB 110] Compacting 1@1 files to L2, score 0.99, subcompactions 1 : 1 2024/11/19-15:18:51.683148 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 108 Base level 1, inputs: [4563(31MB)] 2024/11/19-15:18:51.702367 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-01 2024/11/19-15:18:51.702451 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00104/att-01/compact.done) = No such file or directory: 4.391 sec on http probe failed: 4, retry 2024/11/19-15:18:52.005066 139851137550080 [dcompact_etcd.cc:1163] job-00110/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.321 sec, input = 31.796 MiB 2024/11/19-15:18:52.411791 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-01 2024/11/19-15:18:52.411874 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00106/att-01/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:18:52.737901 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-01 2024/11/19-15:18:52.737975 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00104/att-01/compact.done) = No such file or directory: 5.427 sec on http probe failed: 5, giveup 2024/11/19-15:18:52.737992 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00104/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 51.378 sec, real 5.718 (submit 283.234 ms wait 5.427 s), input 51.378 MiB, send shutdown 2024/11/19-15:18:52.745418 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-01 2024/11/19-15:18:52.745619 139851158521600 [dcompact_etcd.cc:1773] job-00104/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:52.809189 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-00 2024/11/19-15:18:52.809250 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00110/att-00/compact.done) = No such file or directory: 0.795 sec on http probe failed: 1, retry 2024/11/19-15:18:52.898483 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-01 2024/11/19-15:18:52.898569 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00105/att-01/compact.done) = No such file or directory: 1.600 sec on http probe failed: 1, retry 2024/11/19-15:18:52.909346 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-00 2024/11/19-15:18:52.909411 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00108/att-00/compact.done) = No such file or directory: 2.881 sec on http probe failed: 2, retry 2024/11/19-15:18:53.002942 139851158521600 [dcompact_etcd.cc:1163] job-00104/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.257 sec, input = 51.378 MiB 2024/11/19-15:18:53.055180 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-01 2024/11/19-15:18:53.055274 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00106/att-01/compact.done) = No such file or directory: 1.435 sec on http probe failed: 2, retry 2024/11/19-15:18:53.345363 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-00 2024/11/19-15:18:53.345467 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00107/att-00/compact.done) = No such file or directory: 5.039 sec on http probe failed: 2, retry 2024/11/19-15:18:53.453359 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-00 2024/11/19-15:18:53.453422 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00110/att-00/compact.done) = No such file or directory: 1.440 sec on http probe failed: 2, retry 2024/11/19-15:18:53.694286 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-01 2024/11/19-15:18:53.694364 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00106/att-01/compact.done) = No such file or directory: 2.078 sec on http probe failed: 3, retry 2024/11/19-15:18:54.099438 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-00 2024/11/19-15:18:54.099528 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00110/att-00/compact.done) = No such file or directory: 2.084 sec on http probe failed: 3, retry 2024/11/19-15:18:54.187306 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-01 2024/11/19-15:18:54.187410 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00105/att-01/compact.done) = No such file or directory: 2.891 sec on http probe failed: 2, retry 2024/11/19-15:18:54.194883 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-00 2024/11/19-15:18:54.194981 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00108/att-00/compact.done) = No such file or directory: 4.168 sec on http probe failed: 3, retry 2024/11/19-15:18:54.295630 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-02 2024/11/19-15:18:54.295736 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00104/att-02/compact.done) = No such file or directory: 1.285 sec on http probe failed: 1, retry 2024/11/19-15:18:54.335972 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-01 2024/11/19-15:18:54.336046 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00106/att-01/compact.done) = No such file or directory: 2.717 sec on http probe failed: 4, retry 2024/11/19-15:18:54.744432 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-00 2024/11/19-15:18:54.744504 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00110/att-00/compact.done) = No such file or directory: 2.730 sec on http probe failed: 4, retry 2024/11/19-15:18:54.978993 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-01 2024/11/19-15:18:54.979084 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00106/att-01/compact.done) = No such file or directory: 3.359 sec on http probe failed: 5, giveup 2024/11/19-15:18:54.979113 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00106/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.701 sec, real 3.635 (submit 267.117 ms wait 3.359 s), input 31.701 MiB, send shutdown 2024/11/19-15:18:54.988566 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-01 2024/11/19-15:18:55.036036 139851179493120 [dcompact_etcd.cc:1773] job-00106/att-01: CleanFiles 0.047 sec : 0.000 2024/11/19-15:18:55.332251 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-02 2024/11/19-15:18:55.332355 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00104/att-02/compact.done) = No such file or directory: 2.320 sec on http probe failed: 2, retry 2024/11/19-15:18:55.351495 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-00 2024/11/19-15:18:55.351594 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00107/att-00/compact.done) = No such file or directory: 7.048 sec on http probe failed: 3, retry 2024/11/19-15:18:55.388460 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-00 2024/11/19-15:18:55.388549 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00110/att-00/compact.done) = No such file or directory: 3.375 sec on http probe failed: 5, giveup 2024/11/19-15:18:55.388573 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00110/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.796 sec, real 3.705 (submit 321.459 ms wait 3.375 s), input 31.796 MiB, send shutdown 2024/11/19-15:18:55.389615 139851179493120 [dcompact_etcd.cc:1163] job-00106/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.353 sec, input = 31.701 MiB 2024/11/19-15:18:55.396430 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-00 2024/11/19-15:18:55.396680 139851137550080 [dcompact_etcd.cc:1773] job-00110/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:55.475612 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-01 2024/11/19-15:18:55.475690 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00105/att-01/compact.done) = No such file or directory: 4.180 sec on http probe failed: 3, retry 2024/11/19-15:18:55.478917 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-00 2024/11/19-15:18:55.478988 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00108/att-00/compact.done) = No such file or directory: 5.454 sec on http probe failed: 4, retry 2024/11/19-15:18:55.630629 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-01 2024/11/19-15:18:55.630739 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00102/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:18:55.664564 139851137550080 [dcompact_etcd.cc:1163] job-00110/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.268 sec, input = 31.796 MiB 2024/11/19-15:18:56.190542 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-02 2024/11/19-15:18:56.190662 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00106/att-02/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:18:56.365747 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-02 2024/11/19-15:18:56.365843 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00104/att-02/compact.done) = No such file or directory: 3.357 sec on http probe failed: 3, retry 2024/11/19-15:18:56.468671 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-01 2024/11/19-15:18:56.468753 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00110/att-01/compact.done) = No such file or directory: 0.795 sec on http probe failed: 1, retry 2024/11/19-15:18:56.763891 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-01 2024/11/19-15:18:56.763990 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00105/att-01/compact.done) = No such file or directory: 5.469 sec on http probe failed: 4, retry 2024/11/19-15:18:56.764964 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-00 2024/11/19-15:18:56.765027 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00108/att-00/compact.done) = No such file or directory: 6.738 sec on http probe failed: 5, giveup 2024/11/19-15:18:56.765045 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00108/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 63.818 sec, real 7.037 (submit 290.162 ms wait 6.738 s), input 63.818 MiB, send shutdown 2024/11/19-15:18:56.771171 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-00 2024/11/19-15:18:56.771383 139851169007360 [dcompact_etcd.cc:1773] job-00108/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:56.829804 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-02 2024/11/19-15:18:56.829875 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00106/att-02/compact.done) = No such file or directory: 1.435 sec on http probe failed: 2, retry 2024/11/19-15:18:57.070215 139851169007360 [dcompact_etcd.cc:1163] job-00108/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.299 sec, input = 63.818 MiB 2024/11/19-15:18:57.112965 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-01 2024/11/19-15:18:57.113043 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00110/att-01/compact.done) = No such file or directory: 1.440 sec on http probe failed: 2, retry 2024/11/19-15:18:57.360836 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-00 2024/11/19-15:18:57.360938 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00107/att-00/compact.done) = No such file or directory: 9.054 sec on http probe failed: 4, retry 2024/11/19-15:18:57.397907 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-02 2024/11/19-15:18:57.398006 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00104/att-02/compact.done) = No such file or directory: 4.391 sec on http probe failed: 4, retry 2024/11/19-15:18:57.472063 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-02 2024/11/19-15:18:57.472130 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00106/att-02/compact.done) = No such file or directory: 2.074 sec on http probe failed: 3, retry 2024/11/19-15:18:57.639667 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-01 2024/11/19-15:18:57.639740 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00102/att-01/compact.done) = No such file or directory: 7.007 sec on http probe failed: 2, retry 2024/11/19-15:18:57.757061 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-01 2024/11/19-15:18:57.757147 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00110/att-01/compact.done) = No such file or directory: 2.084 sec on http probe failed: 3, retry 2024/11/19-15:18:58.050243 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-01 2024/11/19-15:18:58.050343 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00105/att-01/compact.done) = No such file or directory: 6.757 sec on http probe failed: 5, giveup 2024/11/19-15:18:58.050362 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00105/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 64.000 sec, real 7.057 (submit 294.018 ms wait 6.757 s), input 64.000 MiB, send shutdown 2024/11/19-15:18:58.055363 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-01 2024/11/19-15:18:58.055583 139851127064320 [dcompact_etcd.cc:1773] job-00105/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:58.111004 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-02 2024/11/19-15:18:58.111096 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00106/att-02/compact.done) = No such file or directory: 2.717 sec on http probe failed: 4, retry 2024/11/19-15:18:58.343434 139851127064320 [dcompact_etcd.cc:1163] job-00105/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.288 sec, input = 64.000 MiB 2024/11/19-15:18:58.398415 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-01 2024/11/19-15:18:58.398472 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00110/att-01/compact.done) = No such file or directory: 2.729 sec on http probe failed: 4, retry 2024/11/19-15:18:58.430452 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-02 2024/11/19-15:18:58.430510 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00104/att-02/compact.done) = No such file or directory: 5.423 sec on http probe failed: 5, giveup 2024/11/19-15:18:58.430532 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00104/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 51.378 sec, real 5.685 (submit 257.203 ms wait 5.423 s), input 51.378 MiB, send shutdown 2024/11/19-15:18:58.438856 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00104/att-02 2024/11/19-15:18:58.439077 139851158521600 [dcompact_etcd.cc:1773] job-00104/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:58.440912 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00104) = success 2024/11/19-15:18:58.440926 139851158521600 [dcompact_etcd.cc:1773] job-00104/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:18:58.440948 139851158521600 (Original Log Time 2024/11/19-15:18:41.279568) EVENT_LOG_v1 {"time_micros": 1732000721279561, "job": 104, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4588], "score": 0.89967, "input_data_size": 52342028, "oldest_snapshot_seqno": -1} 2024/11/19-15:18:58.440955 139851158521600 [compaction_job.cc:2437] [default] [JOB 104] Compacting 1@2 files to L3, score 0.90, subcompactions 1 : 1 2024/11/19-15:18:58.440962 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 102 Base level 2, inputs: [4588(49MB)] 2024/11/19-15:18:58.441256 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004629.sst) /storage/mytopling/..//tmp/Topling-82038-gl9A4H 2024/11/19-15:18:58.494535 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004629.sst): first pass time = 0.05's,1011.437'MB/sec 2024/11/19-15:18:58.494540 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004629.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.054 GB 2024/11/19-15:18:58.494572 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004629.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0032 GB, waited 0.000 sec, Key+Value = 0.054 GB 2024/11/19-15:18:58.500832 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004629.sst): index pass time = 0.01's, 154.409'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 80512 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0001 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:18:58.681450 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-01 2024/11/19-15:18:58.681591 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00108/att-01/compact.done) = No such file or directory: 1.597 sec on http probe failed: 1, retry 2024/11/19-15:18:58.760259 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-02 2024/11/19-15:18:58.760350 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00106/att-02/compact.done) = No such file or directory: 3.358 sec on http probe failed: 5, giveup 2024/11/19-15:18:58.760375 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00106/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.701 sec, real 3.724 (submit 353.472 ms wait 3.358 s), input 31.701 MiB, send shutdown 2024/11/19-15:18:58.772180 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00106/att-02 2024/11/19-15:18:58.772436 139851179493120 [dcompact_etcd.cc:1773] job-00106/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:58.778270 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00106) = success 2024/11/19-15:18:58.778285 139851179493120 [dcompact_etcd.cc:1773] job-00106/att-03: CleanFiles 0.006 sec : 0.000 2024/11/19-15:18:58.778311 139851179493120 (Original Log Time 2024/11/19-15:18:47.661651) EVENT_LOG_v1 {"time_micros": 1732000727661643, "job": 106, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [4568], "score": 0.927752, "input_data_size": 33525138, "oldest_snapshot_seqno": 121810407} 2024/11/19-15:18:58.778322 139851179493120 [compaction_job.cc:2437] [default] [JOB 106] Compacting 1@1 files to L2, score 0.93, subcompactions 1 : 1 2024/11/19-15:18:58.778332 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 104 Base level 1, inputs: [4568(31MB)] 2024/11/19-15:18:58.778668 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004630.sst) /storage/mytopling/..//tmp/Topling-82038-TIuSI2 2024/11/19-15:18:58.862065 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004630.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.003 GB, nltTrie workingMem = 0.0002 GB, waited 0.000 sec, Key+Value = 0.033 GB 2024/11/19-15:18:58.865050 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004630.sst): index pass time = 0.00's, 86.980'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 32055 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0003 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.73 2024/11/19-15:18:58.952001 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004630.sst): first pass time = 0.17's, 191.800'MB/sec 2024/11/19-15:18:58.952063 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004630.sst): old prealloc_size = 36877651, real_size = 32764224 2024/11/19-15:18:58.952112 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004630.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 32055 prefix = 4 raw-key = 0.0003 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.73 raw-val = 0.0326 GB zip-val = 0.0327 GB avg-val =1017.01 avg-zval =1020.14 2024/11/19-15:18:58.952198 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004630.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.003 GB, reorder workingMem = 0.0001 GB, waited 0.090 sec, Key+Value = 0.033 GB 2024/11/19-15:18:59.040990 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004630.sst): second pass time = 0.00's,80296233.990'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 368.479'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1200.321'MB/sec (index lex order gen) rebuild zvType time = 0.06's, 0.130'MB/sec write SST data time = 0.03's, 1211.029'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.033'GB zip my value throughput = 80296233.990'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 32055 avg-key = 20.00 avg-zkey = 1.73 avg-val = 1017.01 avg-zval = 1020.14 usrkeys = 32055 avg-key = 12.00 avg-zkey = 1.73 avg-val = 1017.01 avg-zval = 1020.14 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 3996 UnZipSize{ index = 0.0004 GB value = 0.0326 GB dict = 0.00 MB all = 0.0332 GB } __ZipSize{ index = 0.0001 GB value = 0.0327 GB dict = 0.00 MB all = 0.0328 GB } UnZip/Zip{ index = 6.9483 value = 0.9969 dict = 0.00 all = 1.0148 } Zip/UnZip{ index = 0.1439 value = 1.0031 dict = 0.00 all = 0.9854 } ---------------------------- total value len = 4.477196 GB avg = 0.337 KB (by entry num) total key len = 0.335486 GB avg = 0.025 KB total ukey len = 0.229166 GB avg = 0.017 KB total ukey num = 0.013289983 Billion total entry num = 0.013289983 Billion write speed all = 218.036723416 MB/sec (with seq num) write speed all = 213.219941770 MB/sec (without seq num) 2024/11/19-15:18:59.047832 139851179493120 [compaction_job.cc:1948] [default] [JOB 106] Generated table #4630: 32055 keys, 32757444 bytes (need compaction), temperature: kUnknown 2024/11/19-15:18:59.047903 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000739047869, "cf_name": "default", "job": 106, "event": "table_file_creation", "file_number": 4630, "file_size": 32757444, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32700656, "index_size": 55360, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32600271, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000738, "oldest_key_time": 0, "file_creation_time": 1732000738, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4630, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:59.055982 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-01 2024/11/19-15:18:59.056070 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00110/att-01/compact.done) = No such file or directory: 3.372 sec on http probe failed: 5, giveup 2024/11/19-15:18:59.056093 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00110/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.796 sec, real 3.659 (submit 267.767 ms wait 3.372 s), input 31.796 MiB, send shutdown 2024/11/19-15:18:59.065466 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-01 2024/11/19-15:18:59.065702 139851137550080 [dcompact_etcd.cc:1773] job-00110/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:59.071183 139851179493120 [compaction_job.cc:703] job-00106: subcompact[0], size: 0.032757 G, files: /storage/mytopling/.rocksdb [004630] 2024/11/19-15:18:59.075529 139851179493120 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777214 2024/11/19-15:18:59.075677 139851179493120 (Original Log Time 2024/11/19-15:18:58.778369) EVENT_LOG_v1 {"time_micros": 1732000738778349, "job": 106, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [4568], "score": 0.927752, "input_data_size": 33525138, "oldest_snapshot_seqno": 121810407} 2024/11/19-15:18:59.075687 139851179493120 (Original Log Time 2024/11/19-15:18:59.075233) [compaction_job.cc:2022] [default] [JOB 106] Compacted 1@1 files to L2 => 32757444 bytes 2024/11/19-15:18:59.075693 139851179493120 (Original Log Time 2024/11/19-15:18:59.075479) [compaction_job.cc:1224] [default] compacted to: files[6 20 40 3 0 0 0] max score 5.00 (54 files need compaction), MB/sec: 114.5 rd, 111.9 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(32.0, 0.0 +0.0 blob) out(31.2 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 32055, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:59.075713 139851179493120 (Original Log Time 2024/11/19-15:18:59.075520) EVENT_LOG_v1 {"time_micros": 1732000739075496, "cf": "default", "job": 106, "event": "compaction_finished", "compaction_time_micros": 292831, "compaction_time_cpu_micros": 142772, "output_level": 2, "num_output_files": 1, "total_output_size": 32757444, "num_input_records": 32055, "num_output_records": 32055, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 20, 40, 3, 0, 0, 0]} 2024/11/19-15:18:59.075839 139851179493120 [compaction_job.cc:2437] [default] [JOB 111] Compacting 4@2 files to L3, score 1.01, subcompactions 1 : 1 2024/11/19-15:18:59.075852 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 109 Base level 2, inputs: [4590(41MB) 4593(52MB) 4594(52MB) 4597(52MB)] 2024/11/19-15:18:59.375978 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-00 2024/11/19-15:18:59.376093 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00107/att-00/compact.done) = No such file or directory: 11.066 sec on http probe failed: 5, giveup 2024/11/19-15:18:59.376121 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00107/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 121.222 sec, real 11.388 (submit 309.518 ms wait 11.066 s), input 121.222 MiB, send shutdown 2024/11/19-15:18:59.384857 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-00 2024/11/19-15:18:59.385133 139850978166528 [dcompact_etcd.cc:1773] job-00107/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:18:59.409442 139851179493120 [dcompact_etcd.cc:1163] job-00111/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.333 sec, input = 241.761 MiB 2024/11/19-15:18:59.411949 139851137550080 [dcompact_etcd.cc:1163] job-00110/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.346 sec, input = 31.796 MiB 2024/11/19-15:18:59.499786 139851158521600 [top_zip_table_builder.cc:2752] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004629.sst): part-0: reject ZipStore ratio = 0.804 2024/11/19-15:18:59.582842 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004629.sst): old prealloc_size = 57576230, real_size = 52183920 2024/11/19-15:18:59.582946 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004629.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 80512 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0001 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0519 GB zip-val = 0.0520 GB avg-val = 645.14 avg-zval = 646.20 2024/11/19-15:18:59.583044 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004629.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 1.089 sec, Key+Value = 0.054 GB 2024/11/19-15:18:59.648300 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-01 2024/11/19-15:18:59.648403 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00102/att-01/compact.done) = No such file or directory: 9.016 sec on http probe failed: 3, retry 2024/11/19-15:18:59.672709 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004629.sst): second pass time = 1.01's, 51.672'MB/sec, value only(96.4% of KV) wait indexing time = 0.08's, remap KeyValue time = 0.09's, 580.458'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1631.035'MB/sec (index lex order gen) rebuild zvType time = 0.07's, 0.285'MB/sec write SST data time = 0.02's, 2716.050'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = 0.000'MB/sec zip my value time = 1.01's, unzip length = 0.052'GB zip my value throughput = 51.672'MB/sec zip pipeline throughput = 51.672'MB/sec entries = 80512 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.14 avg-zval = 646.20 usrkeys = 80512 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.14 avg-zval = 646.20 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6347 UnZipSize{ index = 0.0013 GB value = 0.0519 GB dict = 0.00 MB all = 0.0539 GB } __ZipSize{ index = 0.0001 GB value = 0.0520 GB dict = 0.00 MB all = 0.0522 GB } UnZip/Zip{ index = 9.4100 value = 0.9984 dict = 0.00 all = 1.0327 } Zip/UnZip{ index = 0.1063 value = 1.0016 dict = 0.00 all = 0.9683 } ---------------------------- total value len = 4.529137 GB avg = 0.339 KB (by entry num) total key len = 0.337418 GB avg = 0.025 KB total ukey len = 0.230454 GB avg = 0.017 KB total ukey num = 0.013370495 Billion total entry num = 0.013370495 Billion write speed all = 214.342797278 MB/sec (with seq num) write speed all = 209.631671036 MB/sec (without seq num) 2024/11/19-15:18:59.675951 139851158521600 [compaction_job.cc:1948] [default] [JOB 104] Generated table #4629: 80512 keys, 52165584 bytes, temperature: kUnknown 2024/11/19-15:18:59.676000 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000739675977, "cf_name": "default", "job": 104, "event": "table_file_creation", "file_number": 4629, "file_size": 52165584, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 52026720, "index_size": 136896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1932288, "raw_average_key_size": 24, "raw_value_size": 51941327, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 80512, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732000738, "oldest_key_time": 0, "file_creation_time": 1732000738, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4629, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:18:59.692415 139850978166528 [dcompact_etcd.cc:1163] job-00107/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.307 sec, input = 121.222 MiB 2024/11/19-15:18:59.692662 139851158521600 [compaction_job.cc:703] job-00104: subcompact[0], size: 0.052166 G, files: /storage/mytopling/.rocksdb [004629] 2024/11/19-15:18:59.693261 139851158521600 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777216 2024/11/19-15:18:59.693366 139851158521600 (Original Log Time 2024/11/19-15:18:58.440993) EVENT_LOG_v1 {"time_micros": 1732000738440974, "job": 104, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4588], "score": 0.89967, "input_data_size": 52342028, "oldest_snapshot_seqno": -1} 2024/11/19-15:18:59.693373 139851158521600 (Original Log Time 2024/11/19-15:18:59.693073) [compaction_job.cc:2022] [default] [JOB 104] Compacted 1@2 files to L3 => 52165584 bytes 2024/11/19-15:18:59.693376 139851158521600 (Original Log Time 2024/11/19-15:18:59.693231) [compaction_job.cc:1224] [default] compacted to: files[6 20 39 4 0 0 0] max score 5.00 (50 files need compaction), MB/sec: 41.8 rd, 41.7 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(49.9, 0.0 +0.0 blob) out(49.7 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 80512, records dropped: 0 output_compression: Snappy 2024/11/19-15:18:59.693391 139851158521600 (Original Log Time 2024/11/19-15:18:59.693255) EVENT_LOG_v1 {"time_micros": 1732000739693243, "cf": "default", "job": 104, "event": "compaction_finished", "compaction_time_micros": 1251681, "compaction_time_cpu_micros": 453778, "output_level": 3, "num_output_files": 1, "total_output_size": 52165584, "num_input_records": 80512, "num_output_records": 80512, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 20, 39, 4, 0, 0, 0]} 2024/11/19-15:18:59.693453 139851158521600 [compaction_job.cc:2437] [default] [JOB 112] Compacting 1@2 files to L3, score 0.86, subcompactions 1 : 1 2024/11/19-15:18:59.693461 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 110 Base level 2, inputs: [4622(63MB)] 2024/11/19-15:18:59.949237 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-02 2024/11/19-15:18:59.949345 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00105/att-02/compact.done) = No such file or directory: 1.600 sec on http probe failed: 1, retry 2024/11/19-15:18:59.969931 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-01 2024/11/19-15:18:59.969993 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00108/att-01/compact.done) = No such file or directory: 2.888 sec on http probe failed: 2, retry 2024/11/19-15:19:00.012301 139851158521600 [dcompact_etcd.cc:1163] job-00112/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.318 sec, input = 64.001 MiB 2024/11/19-15:19:00.213187 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-02 2024/11/19-15:19:00.213290 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00110/att-02/compact.done) = No such file or directory: 0.795 sec on http probe failed: 1, retry 2024/11/19-15:19:00.857102 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-02 2024/11/19-15:19:00.857205 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00110/att-02/compact.done) = No such file or directory: 1.437 sec on http probe failed: 2, retry 2024/11/19-15:19:01.238264 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-02 2024/11/19-15:19:01.238362 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00105/att-02/compact.done) = No such file or directory: 2.886 sec on http probe failed: 2, retry 2024/11/19-15:19:01.251012 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-01 2024/11/19-15:19:01.251083 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00108/att-01/compact.done) = No such file or directory: 4.176 sec on http probe failed: 3, retry 2024/11/19-15:19:01.497983 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-02 2024/11/19-15:19:01.498074 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00110/att-02/compact.done) = No such file or directory: 2.081 sec on http probe failed: 3, retry 2024/11/19-15:19:01.618230 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-00 2024/11/19-15:19:01.618311 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00112/att-00/compact.done) = No such file or directory: 1.600 sec on http probe failed: 1, retry 2024/11/19-15:19:01.653664 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-01 2024/11/19-15:19:01.653728 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00102/att-01/compact.done) = No such file or directory: 11.024 sec on http probe failed: 4, retry 2024/11/19-15:19:02.143140 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-02 2024/11/19-15:19:02.143231 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00110/att-02/compact.done) = No such file or directory: 2.722 sec on http probe failed: 4, retry 2024/11/19-15:19:02.527718 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-02 2024/11/19-15:19:02.527808 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00105/att-02/compact.done) = No such file or directory: 4.175 sec on http probe failed: 3, retry 2024/11/19-15:19:02.533167 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-01 2024/11/19-15:19:02.533240 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00108/att-01/compact.done) = No such file or directory: 5.457 sec on http probe failed: 4, retry 2024/11/19-15:19:02.732943 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-01 2024/11/19-15:19:02.733055 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00107/att-01/compact.done) = No such file or directory: 3.031 sec on http probe failed: 1, retry 2024/11/19-15:19:02.787904 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-02 2024/11/19-15:19:02.787979 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00110/att-02/compact.done) = No such file or directory: 3.367 sec on http probe failed: 5, giveup 2024/11/19-15:19:02.787995 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00110/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.796 sec, real 3.722 (submit 346.124 ms wait 3.367 s), input 31.796 MiB, send shutdown 2024/11/19-15:19:02.796210 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00110/att-02 2024/11/19-15:19:02.796428 139851137550080 [dcompact_etcd.cc:1773] job-00110/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:02.798299 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00110) = success 2024/11/19-15:19:02.798307 139851137550080 [dcompact_etcd.cc:1773] job-00110/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:19:02.798326 139851137550080 (Original Log Time 2024/11/19-15:18:51.683164) EVENT_LOG_v1 {"time_micros": 1732000731683155, "job": 110, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [4563], "score": 0.993214, "input_data_size": 33524882, "oldest_snapshot_seqno": 121919566} 2024/11/19-15:19:02.798333 139851137550080 [compaction_job.cc:2437] [default] [JOB 110] Compacting 1@1 files to L2, score 0.99, subcompactions 1 : 1 2024/11/19-15:19:02.798342 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 108 Base level 1, inputs: [4563(31MB)] 2024/11/19-15:19:02.798626 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004631.sst) /storage/mytopling/..//tmp/Topling-82038-doWFYR 2024/11/19-15:19:02.825377 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004631.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.033 GB 2024/11/19-15:19:02.825528 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004631.sst): index pass time = 0.00's,2157.465'MB/sec index type = UintIndex_AllOne usrkeys = 32150 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 6 raw-key = 0.0003 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.01 2024/11/19-15:19:02.858520 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004631.sst): first pass time = 0.06's, 556.767'MB/sec 2024/11/19-15:19:02.858568 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004631.sst): old prealloc_size = 36877370, real_size = 32806568 2024/11/19-15:19:02.858590 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004631.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 32150 prefix = 4 raw-key = 0.0003 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.01 raw-val = 0.0327 GB zip-val = 0.0328 GB avg-val =1017.03 avg-zval =1020.16 2024/11/19-15:19:02.885028 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004631.sst): second pass time = 0.00's,91079409.471'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.03's, 1240.228'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.759'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 0.427'MB/sec write SST data time = 0.01's, 4311.523'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.033'GB zip my value throughput = 91079409.471'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 32150 avg-key = 20.00 avg-zkey = 0.01 avg-val = 1017.03 avg-zval = 1020.16 usrkeys = 32150 avg-key = 12.00 avg-zkey = 0.01 avg-val = 1017.03 avg-zval = 1020.16 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0004 GB value = 0.0327 GB dict = 0.00 MB all = 0.0333 GB } __ZipSize{ index = 0.0000 GB value = 0.0328 GB dict = 0.00 MB all = 0.0328 GB } UnZip/Zip{ index =2009.3750 value = 0.9969 dict = 0.00 all = 1.0165 } Zip/UnZip{ index = 0.0005 value = 1.0031 dict = 0.00 all = 0.9838 } ---------------------------- total value len = 4.561834 GB avg = 0.340 KB (by entry num) total key len = 0.338061 GB avg = 0.025 KB total ukey len = 0.230840 GB avg = 0.017 KB total ukey num = 0.013402645 Billion total entry num = 0.013402645 Billion write speed all = 189.062082027 MB/sec (with seq num) write speed all = 184.924961917 MB/sec (without seq num) 2024/11/19-15:19:02.886902 139851137550080 [compaction_job.cc:1948] [default] [JOB 110] Generated table #4631: 32150 keys, 32799498 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:02.886963 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000742886939, "cf_name": "default", "job": 110, "event": "table_file_creation", "file_number": 4631, "file_size": 32799498, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32798192, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32697508, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000742, "oldest_key_time": 0, "file_creation_time": 1732000742, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4631, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:02.894012 139851137550080 [compaction_job.cc:703] job-00110: subcompact[0], size: 0.032799 G, files: /storage/mytopling/.rocksdb [004631] 2024/11/19-15:19:02.894470 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777216 2024/11/19-15:19:02.894578 139851137550080 (Original Log Time 2024/11/19-15:19:02.798371) EVENT_LOG_v1 {"time_micros": 1732000742798356, "job": 110, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [4563], "score": 0.993214, "input_data_size": 33524882, "oldest_snapshot_seqno": 121919566} 2024/11/19-15:19:02.894584 139851137550080 (Original Log Time 2024/11/19-15:19:02.894264) [compaction_job.cc:2022] [default] [JOB 110] Compacted 1@1 files to L2 => 32799498 bytes 2024/11/19-15:19:02.894587 139851137550080 (Original Log Time 2024/11/19-15:19:02.894441) [compaction_job.cc:1224] [default] compacted to: files[6 19 40 4 0 0 0] max score 5.28 (50 files need compaction), MB/sec: 350.5 rd, 342.9 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(32.0, 0.0 +0.0 blob) out(31.3 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 32150, records dropped: 0 output_compression: Snappy 2024/11/19-15:19:02.894602 139851137550080 (Original Log Time 2024/11/19-15:19:02.894464) EVENT_LOG_v1 {"time_micros": 1732000742894452, "cf": "default", "job": 110, "event": "compaction_finished", "compaction_time_micros": 95652, "compaction_time_cpu_micros": 95217, "output_level": 2, "num_output_files": 1, "total_output_size": 32799498, "num_input_records": 32150, "num_output_records": 32150, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 19, 40, 4, 0, 0, 0]} 2024/11/19-15:19:02.902797 139851137550080 [compaction_job.cc:2437] [default] [JOB 113] Compacting 6@0 + 19@1 files to L1, score 5.28, subcompactions 3 : 3 2024/11/19-15:19:02.902815 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 111 Base level 0, inputs: [4627(512MB) 4605(516MB) 4596(511MB) 4586(520MB) 4579(516MB) 4574(512MB)], [4548(31MB) 4549(31MB) 4550(31MB) 4551(31MB) 4553(31MB) 4555(31MB) 4556(31MB) 4557(31MB) 4558(31MB) 4559(31MB) 4560(31MB) 4561(31MB) 4562(31MB) 4565(31MB) 4566(31MB) 4569(31MB) 4570(31MB) 4571(31MB) 4572(10115KB)] 2024/11/19-15:19:02.903474 139851137550080 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4634: 16 keys, 6131 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:02.903509 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000742903490, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4634, "file_size": 6131, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 114603865, "largest_seqno": 121760399, "table_properties": {"data_size": 1984, "index_size": 3328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 256, "raw_average_key_size": 16, "raw_value_size": 1932, "raw_average_value_size": 120, "num_data_blocks": 1, "num_entries": 16, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000742, "oldest_key_time": 0, "file_creation_time": 1732000742, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4634, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:02.904138 139851137550080 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4635: 161 keys, 28024 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:02.904166 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000742904151, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4635, "file_size": 28024, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 113825375, "largest_seqno": 121850416, "table_properties": {"data_size": 19584, "index_size": 7616, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3220, "raw_average_key_size": 20, "raw_value_size": 19526, "raw_average_value_size": 121, "num_data_blocks": 1, "num_entries": 161, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000742, "oldest_key_time": 0, "file_creation_time": 1732000742, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4635, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:02.904303 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-00 2024/11/19-15:19:02.904383 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00112/att-00/compact.done) = No such file or directory: 2.886 sec on http probe failed: 2, retry 2024/11/19-15:19:02.941909 139851137550080 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4636: 49840 keys, 33506880 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:02.941962 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000742941942, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4636, "file_size": 33506880, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 113835379, "largest_seqno": 114527092, "table_properties": {"data_size": 32159360, "index_size": 1346688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196160, "raw_average_key_size": 24, "raw_value_size": 32159348, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49840, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000742, "oldest_key_time": 0, "file_creation_time": 1732000742, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4636, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:02.979870 139851137550080 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4637: 49870 keys, 33506688 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:02.979928 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000742979899, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4637, "file_size": 33506688, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 114485951, "largest_seqno": 115522995, "table_properties": {"data_size": 32156800, "index_size": 1349056, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196880, "raw_average_key_size": 24, "raw_value_size": 32156779, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49870, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000742, "oldest_key_time": 0, "file_creation_time": 1732000742, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4637, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.007048 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4633: 231460 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.007113 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000743007092, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4633, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 117178976, "largest_seqno": 117749063, "table_properties": {"data_size": 27025984, "index_size": 6298176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480880, "raw_average_key_size": 28, "raw_value_size": 27025964, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231460, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000742, "oldest_key_time": 0, "file_creation_time": 1732000742, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4633, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.017278 139851137550080 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4638: 49849 keys, 33506560 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.017321 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000743017301, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4638, "file_size": 33506560, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 115274201, "largest_seqno": 116519791, "table_properties": {"data_size": 32158720, "index_size": 1347008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196376, "raw_average_key_size": 24, "raw_value_size": 32158716, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49849, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000742, "oldest_key_time": 0, "file_creation_time": 1732000742, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4638, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.049392 139849738749696 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4632: 422215 keys, 33134211 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.049475 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000743049450, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4632, "file_size": 33134211, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 114288414, "largest_seqno": 121311341, "table_properties": {"data_size": 22799680, "index_size": 10333696, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 8444300, "raw_average_key_size": 20, "raw_value_size": 22799646, "raw_average_value_size": 54, "num_data_blocks": 1, "num_entries": 422215, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000742, "oldest_key_time": 0, "file_creation_time": 1732000742, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4632, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.054968 139851137550080 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4640: 49829 keys, 33506688 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.055013 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000743054992, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4640, "file_size": 33506688, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 116032573, "largest_seqno": 117271357, "table_properties": {"data_size": 32158464, "index_size": 1347392, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1195896, "raw_average_key_size": 24, "raw_value_size": 32158431, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49829, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4640, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.058741 139849738749696 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4641: 26708 keys, 2127805 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.058782 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000743058763, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4641, "file_size": 2127805, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121311342, "largest_seqno": 121696676, "table_properties": {"data_size": 1442624, "index_size": 684352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 534160, "raw_average_key_size": 20, "raw_value_size": 1442599, "raw_average_value_size": 54, "num_data_blocks": 1, "num_entries": 26708, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4641, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.164969 139851137550080 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4642: 49842 keys, 33506880 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.165027 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000743165003, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4642, "file_size": 33506880, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 117036464, "largest_seqno": 118028868, "table_properties": {"data_size": 32159808, "index_size": 1346240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196208, "raw_average_key_size": 24, "raw_value_size": 32159760, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49842, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4642, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.180848 139849738749696 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4643: 144000 keys, 3763278 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.180897 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000743180876, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4643, "file_size": 3763278, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 113894664, "largest_seqno": 121750395, "table_properties": {"data_size": 144000, "index_size": 3618432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3456000, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 144000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4643, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.180962 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4639: 231496 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.181006 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000743180986, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4639, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 117749064, "largest_seqno": 118368350, "table_properties": {"data_size": 27027840, "index_size": 6296320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481888, "raw_average_key_size": 28, "raw_value_size": 27027800, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231496, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4639, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.202540 139851137550080 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4644: 49900 keys, 33507008 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.202589 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000743202568, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4644, "file_size": 33507008, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 118028869, "largest_seqno": 118719318, "table_properties": {"data_size": 32157120, "index_size": 1349056, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1197600, "raw_average_key_size": 24, "raw_value_size": 32157099, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49900, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4644, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.256759 139851137550080 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4647: 49881 keys, 33507072 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.256814 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000743256792, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4647, "file_size": 33507072, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 118719319, "largest_seqno": 119513401, "table_properties": {"data_size": 32158144, "index_size": 1348096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1197144, "raw_average_key_size": 24, "raw_value_size": 32158096, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49881, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4647, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.296442 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4646: 231284 keys, 33325123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.296497 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000743296475, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4646, "file_size": 33325123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 118077628, "largest_seqno": 118555809, "table_properties": {"data_size": 27031232, "index_size": 6293056, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6475952, "raw_average_key_size": 28, "raw_value_size": 27031200, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231284, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4646, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.307077 139851137550080 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4648: 49835 keys, 33507008 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.307126 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000743307105, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4648, "file_size": 33507008, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119513402, "largest_seqno": 120512281, "table_properties": {"data_size": 32159808, "index_size": 1346368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196040, "raw_average_key_size": 24, "raw_value_size": 32159790, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49835, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4648, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.357640 139852362282752 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16273899 2024/11/19-15:19:03.365121 139851137550080 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4650: 49846 keys, 33506624 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.365166 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000743365145, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4650, "file_size": 33506624, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120186943, "largest_seqno": 121542540, "table_properties": {"data_size": 32158464, "index_size": 1347328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196304, "raw_average_key_size": 24, "raw_value_size": 32158452, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49846, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4650, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.376070 139849738749696 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4645: 480000 keys, 21273859 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.376118 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000743376098, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4645, "file_size": 21273859, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 113841381, "largest_seqno": 121699677, "table_properties": {"data_size": 9504000, "index_size": 11769024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 11520000, "raw_average_key_size": 24, "raw_value_size": 9504000, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 480000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4645, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.406738 139851137550080 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4651: 34308 keys, 23098496 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.406786 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000743406766, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4651, "file_size": 23098496, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120989058, "largest_seqno": 121863419, "table_properties": {"data_size": 22126784, "index_size": 970880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 823392, "raw_average_key_size": 24, "raw_value_size": 22126734, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 34308, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4651, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.417310 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4649: 231410 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.417354 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000743417334, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4649, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 118420486, "largest_seqno": 119001217, "table_properties": {"data_size": 27028544, "index_size": 6295680, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6479480, "raw_average_key_size": 28, "raw_value_size": 27028500, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231410, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4649, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.462852 139851137550080 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4653: 31077 keys, 2574653 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.462895 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000743462875, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4653, "file_size": 2574653, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 113838380, "largest_seqno": 114288413, "table_properties": {"data_size": 1677568, "index_size": 896256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 621540, "raw_average_key_size": 20, "raw_value_size": 1677525, "raw_average_value_size": 53, "num_data_blocks": 1, "num_entries": 31077, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4653, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.579227 139849738749696 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4652: 231249 keys, 33325123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.579277 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000743579256, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4652, "file_size": 33325123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 113854383, "largest_seqno": 114433484, "table_properties": {"data_size": 27035072, "index_size": 6289216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6474972, "raw_average_key_size": 28, "raw_value_size": 27035066, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231249, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4652, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.590817 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4654: 231452 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.590863 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000743590843, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4654, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119001218, "largest_seqno": 119554212, "table_properties": {"data_size": 27023616, "index_size": 6300544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480656, "raw_average_key_size": 28, "raw_value_size": 27023588, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231452, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4654, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.662351 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-01 2024/11/19-15:19:03.662435 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00102/att-01/compact.done) = No such file or directory: 13.030 sec on http probe failed: 5, giveup 2024/11/19-15:19:03.662455 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00102/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 254.856 sec, real 13.340 (submit 301.697 ms wait 13.030 s), input 254.856 MiB, send shutdown 2024/11/19-15:19:03.671316 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-01 2024/11/19-15:19:03.671540 139851148035840 [dcompact_etcd.cc:1773] job-00102/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:03.683263 139849738749696 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4655: 231481 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.683309 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000743683288, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4655, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 114050673, "largest_seqno": 114602961, "table_properties": {"data_size": 27026752, "index_size": 6297408, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481468, "raw_average_key_size": 28, "raw_value_size": 27026690, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231481, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4655, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.689158 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4656: 231424 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.689202 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000743689181, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4656, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119085336, "largest_seqno": 119642715, "table_properties": {"data_size": 27028480, "index_size": 6295744, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6479872, "raw_average_key_size": 28, "raw_value_size": 27028476, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231424, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4656, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.796395 139849738749696 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4657: 231455 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.796456 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000743796434, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4657, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 114461920, "largest_seqno": 115089612, "table_properties": {"data_size": 27023488, "index_size": 6300736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480740, "raw_average_key_size": 28, "raw_value_size": 27023474, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231455, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4657, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.812384 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-02 2024/11/19-15:19:03.812467 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00105/att-02/compact.done) = No such file or directory: 5.464 sec on http probe failed: 4, retry 2024/11/19-15:19:03.813835 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-01 2024/11/19-15:19:03.813895 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00108/att-01/compact.done) = No such file or directory: 6.739 sec on http probe failed: 5, giveup 2024/11/19-15:19:03.813911 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00108/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 63.818 sec, real 7.042 (submit 298.750 ms wait 6.739 s), input 63.818 MiB, send shutdown 2024/11/19-15:19:03.818486 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-01 2024/11/19-15:19:03.847095 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4658: 231279 keys, 33325187 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.847154 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000743847124, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4658, "file_size": 33325187, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119642716, "largest_seqno": 120251836, "table_properties": {"data_size": 27030912, "index_size": 6293440, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6475812, "raw_average_key_size": 28, "raw_value_size": 27030874, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231279, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4658, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.847159 139851169007360 [dcompact_etcd.cc:1773] job-00108/att-01: CleanFiles 0.029 sec : 0.000 2024/11/19-15:19:03.899752 139849738749696 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4659: 231323 keys, 33325187 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.899800 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000743899780, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4659, "file_size": 33325187, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 115052386, "largest_seqno": 115592960, "table_properties": {"data_size": 27033024, "index_size": 6291328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6477044, "raw_average_key_size": 28, "raw_value_size": 27032998, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231323, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4659, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.950741 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4660: 231475 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:03.950792 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000743950770, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4660, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120043193, "largest_seqno": 120582684, "table_properties": {"data_size": 27025984, "index_size": 6298176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481300, "raw_average_key_size": 28, "raw_value_size": 27025934, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231475, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4660, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:03.957178 139851148035840 [dcompact_etcd.cc:1163] job-00102/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.286 sec, input = 254.856 MiB 2024/11/19-15:19:04.011967 139849738749696 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4661: 231399 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.012017 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000744011996, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4661, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 115076073, "largest_seqno": 115697833, "table_properties": {"data_size": 27027072, "index_size": 6297152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6479172, "raw_average_key_size": 28, "raw_value_size": 27027050, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231399, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4661, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.121608 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4662: 231437 keys, 33324931 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.121664 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000744121641, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4662, "file_size": 33324931, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120292839, "largest_seqno": 120944755, "table_properties": {"data_size": 27023936, "index_size": 6300160, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480236, "raw_average_key_size": 28, "raw_value_size": 27023914, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231437, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000743, "oldest_key_time": 0, "file_creation_time": 1732000743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4662, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.136296 139851169007360 [dcompact_etcd.cc:1163] job-00108/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 63.818 MiB 2024/11/19-15:19:04.169650 139849738749696 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4663: 231443 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.169713 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000744169682, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4663, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 115697834, "largest_seqno": 116328166, "table_properties": {"data_size": 27024128, "index_size": 6300032, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480404, "raw_average_key_size": 28, "raw_value_size": 27024086, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231443, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4663, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.202911 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-00 2024/11/19-15:19:04.202984 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00112/att-00/compact.done) = No such file or directory: 4.172 sec on http probe failed: 3, retry 2024/11/19-15:19:04.221790 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4664: 231304 keys, 33325187 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.221839 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000744221816, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4664, "file_size": 33325187, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120944756, "largest_seqno": 121503416, "table_properties": {"data_size": 27029760, "index_size": 6294592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6476512, "raw_average_key_size": 28, "raw_value_size": 27029704, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231304, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4664, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.273169 139849738749696 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4665: 231454 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.273222 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000744273201, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4665, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 116050580, "largest_seqno": 116590026, "table_properties": {"data_size": 27026880, "index_size": 6297344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480712, "raw_average_key_size": 28, "raw_value_size": 27026828, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231454, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4665, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.329249 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4666: 231395 keys, 33325123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.329304 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000744329281, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4666, "file_size": 33325123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121031891, "largest_seqno": 121588983, "table_properties": {"data_size": 27028416, "index_size": 6295872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6479060, "raw_average_key_size": 28, "raw_value_size": 27028398, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231395, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4666, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.418285 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-00 2024/11/19-15:19:04.418437 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00111/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:19:04.433398 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4668: 103685 keys, 14990339 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.433457 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000744433434, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4668, "file_size": 14990339, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121525138, "largest_seqno": 121739492, "table_properties": {"data_size": 12091712, "index_size": 2897792, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2903180, "raw_average_key_size": 28, "raw_value_size": 12091690, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 103685, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4668, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.442619 139849738749696 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4667: 231262 keys, 33325123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.442667 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000744442646, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4667, "file_size": 33325123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 116318931, "largest_seqno": 116949926, "table_properties": {"data_size": 27033984, "index_size": 6290304, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6475336, "raw_average_key_size": 28, "raw_value_size": 27033944, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231262, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4667, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.464769 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4669: 32078 keys, 33524370 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.464818 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000744464797, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4669, "file_size": 33524370, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32624256, "index_size": 899264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641560, "raw_average_key_size": 20, "raw_value_size": 32624241, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32078, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4669, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.591095 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4671: 32057 keys, 33525202 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.591149 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000744591128, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4671, "file_size": 33525202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32603008, "index_size": 921344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602951, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4671, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.591175 139849738749696 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4670: 231462 keys, 33324931 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.591219 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000744591200, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4670, "file_size": 33324931, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 116949927, "largest_seqno": 117538943, "table_properties": {"data_size": 27025984, "index_size": 6298112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480936, "raw_average_key_size": 28, "raw_value_size": 27025928, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231462, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4670, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.622182 139849738749696 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4673: 67858 keys, 9880000 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.622232 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000744622212, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4673, "file_size": 9880000, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 117079301, "largest_seqno": 117557015, "table_properties": {"data_size": 7935232, "index_size": 1943936, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1900024, "raw_average_key_size": 28, "raw_value_size": 7935192, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 67858, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4673, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.622324 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4672: 32060 keys, 33524754 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.622368 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000744622348, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4672, "file_size": 33524754, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32603904, "index_size": 920000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32603898, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4672, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.666441 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4674: 32111 keys, 33524626 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.666493 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000744666470, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4674, "file_size": 33524626, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32655232, "index_size": 868544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642220, "raw_average_key_size": 20, "raw_value_size": 32655214, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32111, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4674, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.741295 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-01 2024/11/19-15:19:04.741378 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00107/att-01/compact.done) = No such file or directory: 5.041 sec on http probe failed: 2, retry 2024/11/19-15:19:04.768320 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4675: 32061 keys, 33525202 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.768366 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000744768344, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4675, "file_size": 33525202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32603520, "index_size": 920832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641220, "raw_average_key_size": 20, "raw_value_size": 32603491, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32061, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4675, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.800019 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4676: 32057 keys, 33524434 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.800071 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000744800049, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4676, "file_size": 33524434, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32601792, "index_size": 921792, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601780, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4676, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.831518 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4677: 32058 keys, 33525074 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.831568 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000744831548, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4677, "file_size": 33525074, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32603840, "index_size": 920384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603779, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4677, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.871645 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4678: 32150 keys, 33524242 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.871696 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000744871676, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4678, "file_size": 33524242, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32696960, "index_size": 826432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32696925, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4678, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.917227 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4679: 32055 keys, 33524562 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.917277 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000744917257, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4679, "file_size": 33524562, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32601280, "index_size": 922432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32601265, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4679, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.956979 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4680: 32057 keys, 33524882 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.957030 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000744957009, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4680, "file_size": 33524882, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32601152, "index_size": 922880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601105, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4680, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:04.995591 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4681: 32126 keys, 33525202 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:04.995640 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000744995619, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4681, "file_size": 33525202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32673472, "index_size": 850880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642520, "raw_average_key_size": 20, "raw_value_size": 32673435, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32126, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4681, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:05.026276 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4682: 32057 keys, 33525138 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:05.026323 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000745026303, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4682, "file_size": 33525138, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32602368, "index_size": 921920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602308, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000744, "oldest_key_time": 0, "file_creation_time": 1732000744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4682, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:05.064646 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4683: 32050 keys, 33525330 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:05.064691 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000745064671, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4683, "file_size": 33525330, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32597248, "index_size": 927232, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641000, "raw_average_key_size": 20, "raw_value_size": 32597197, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32050, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4683, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:05.101387 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-02 2024/11/19-15:19:05.101493 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00105/att-02/compact.done) = No such file or directory: 6.749 sec on http probe failed: 5, giveup 2024/11/19-15:19:05.101512 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00105/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 64.000 sec, real 7.046 (submit 287.761 ms wait 6.749 s), input 64.000 MiB, send shutdown 2024/11/19-15:19:05.106928 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4684: 32055 keys, 33525074 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:05.106972 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000745106952, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4684, "file_size": 33525074, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32599872, "index_size": 924352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32599861, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4684, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:05.110545 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00105/att-02 2024/11/19-15:19:05.127310 139851127064320 [dcompact_etcd.cc:1773] job-00105/att-02: CleanFiles 0.017 sec : 0.000 2024/11/19-15:19:05.148436 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4685: 32059 keys, 33524434 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:05.148479 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000745148459, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4685, "file_size": 33524434, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32604544, "index_size": 919040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32604512, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4685, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:05.155253 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00105) = success 2024/11/19-15:19:05.155267 139851127064320 [dcompact_etcd.cc:1773] job-00105/att-03: CleanFiles 0.028 sec : 0.000 2024/11/19-15:19:05.155288 139851127064320 (Original Log Time 2024/11/19-15:18:43.909893) EVENT_LOG_v1 {"time_micros": 1732000723909887, "job": 105, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4589], "score": 0.886859, "input_data_size": 55501528, "oldest_snapshot_seqno": 121750395} 2024/11/19-15:19:05.155295 139851127064320 [compaction_job.cc:2437] [default] [JOB 105] Compacting 1@2 files to L3, score 0.89, subcompactions 1 : 1 2024/11/19-15:19:05.155302 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 103 Base level 2, inputs: [4589(52MB)] 2024/11/19-15:19:05.155542 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004687.sst) /storage/mytopling/..//tmp/Topling-82038-ZdKuKe 2024/11/19-15:19:05.198443 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4686: 32061 keys, 33525202 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:05.198490 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000745198469, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4686, "file_size": 33525202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32605312, "index_size": 919040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641220, "raw_average_key_size": 20, "raw_value_size": 32605260, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32061, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4686, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:05.244883 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4688: 32052 keys, 33524946 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:05.244944 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000745244915, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4688, "file_size": 33524946, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32598720, "index_size": 925376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641040, "raw_average_key_size": 20, "raw_value_size": 32598694, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32052, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4688, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:05.285044 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4689: 32148 keys, 33525010 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:05.285096 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000745285075, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4689, "file_size": 33525010, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32695424, "index_size": 828736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642960, "raw_average_key_size": 20, "raw_value_size": 32695396, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32148, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4689, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:05.324770 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4690: 32143 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:05.324821 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000745324801, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4690, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 113886912, "table_properties": {"data_size": 32688128, "index_size": 836032, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642860, "raw_average_key_size": 20, "raw_value_size": 32688092, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32143, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4690, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:05.330459 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004687.sst): first pass time = 0.17's, 383.685'MB/sec 2024/11/19-15:19:05.330465 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004687.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:19:05.330486 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004687.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.006 GB, dictZip workingMem = 0.0033 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:19:05.366222 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004687.sst): index pass time = 0.04's, 207.514'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463425 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:19:05.366979 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4691: 32149 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:05.367030 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000745367008, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4691, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 113886913, "largest_seqno": 113928971, "table_properties": {"data_size": 32697280, "index_size": 826368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642980, "raw_average_key_size": 20, "raw_value_size": 32697259, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32149, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4691, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:05.402884 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4692: 32057 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:05.402944 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000745402917, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4692, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 113928972, "largest_seqno": 113998063, "table_properties": {"data_size": 32603200, "index_size": 921152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32603144, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4692, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:05.471426 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4693: 32060 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:05.471478 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000745471456, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4693, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 113998064, "largest_seqno": 114702420, "table_properties": {"data_size": 32602816, "index_size": 920832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32602766, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4693, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:05.488649 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-00 2024/11/19-15:19:05.488738 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00112/att-00/compact.done) = No such file or directory: 5.471 sec on http probe failed: 4, retry 2024/11/19-15:19:05.574363 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4694: 32136 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:05.574434 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000745574401, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4694, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 114702421, "largest_seqno": 114785724, "table_properties": {"data_size": 32683200, "index_size": 840512, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642720, "raw_average_key_size": 20, "raw_value_size": 32683162, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32136, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4694, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:05.608978 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4695: 32058 keys, 33524800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:05.609055 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000745609022, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4695, "file_size": 33524800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 114785725, "largest_seqno": 114847791, "table_properties": {"data_size": 32603136, "index_size": 920832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603088, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4695, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:05.746423 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-02 2024/11/19-15:19:05.746550 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00108/att-02/compact.done) = No such file or directory: 1.597 sec on http probe failed: 1, retry 2024/11/19-15:19:05.794755 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4696: 32057 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:05.794834 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000745794804, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4696, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 114775072, "largest_seqno": 114855081, "table_properties": {"data_size": 32603328, "index_size": 920448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32603311, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4696, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:05.925735 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4697: 32148 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:05.925847 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000745925801, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4697, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 114819845, "largest_seqno": 114881004, "table_properties": {"data_size": 32697536, "index_size": 826624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642960, "raw_average_key_size": 20, "raw_value_size": 32697495, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32148, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4697, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:05.976342 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4698: 32058 keys, 33524352 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:05.976403 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000745976380, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4698, "file_size": 33524352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 114881005, "largest_seqno": 114983473, "table_properties": {"data_size": 32601984, "index_size": 921536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32601949, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4698, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:06.085165 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004687.sst): old prealloc_size = 61051680, real_size = 18323504 2024/11/19-15:19:06.085415 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004687.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 463425 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0173 GB avg-val = 116.81 avg-zval = 37.40 2024/11/19-15:19:06.085479 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004687.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.755 sec, Key+Value = 0.067 GB 2024/11/19-15:19:06.090416 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4699: 32059 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:06.090467 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000746090445, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4699, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 114983474, "largest_seqno": 115660391, "table_properties": {"data_size": 32604352, "index_size": 920128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32604295, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4699, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:06.192487 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4700: 32091 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:06.192539 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000746192518, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4700, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 115660392, "largest_seqno": 115735354, "table_properties": {"data_size": 32636736, "index_size": 887296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641820, "raw_average_key_size": 20, "raw_value_size": 32636710, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32091, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000746, "oldest_key_time": 0, "file_creation_time": 1732000746, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4700, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:06.294554 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4701: 32060 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:06.294608 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000746294586, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4701, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 115735355, "largest_seqno": 115818405, "table_properties": {"data_size": 32603008, "index_size": 921408, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32602996, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000746, "oldest_key_time": 0, "file_creation_time": 1732000746, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4701, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:06.350607 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004687.sst): second pass time = 0.75's, 71.733'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.27's, 70.671'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3720.352'MB/sec (index lex order gen) rebuild zvType time = 0.22's, 0.534'MB/sec write SST data time = 0.05's, 390.120'MB/sec dict compress time = 0.00's, 1291.821'MB/sec z-dict build time = 0.00's, sample length = 0.545'MB, throughput = 9232.510'MB/sec zip my value time = 0.75's, unzip length = 0.054'GB zip my value throughput = 71.733'MB/sec zip pipeline throughput = 71.733'MB/sec entries = 463425 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 37.40 usrkeys = 463425 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 37.40 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.55 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0173 GB dict = 0.55 MB all = 0.0188 GB } UnZip/Zip{ index = 10.5693 value = 3.1236 dict = 1.00 all = 3.5776 } Zip/UnZip{ index = 0.0946 value = 0.3201 dict = 1.00 all = 0.2795 } ---------------------------- total value len = 4.615967 GB avg = 0.333 KB (by entry num) total key len = 0.351037 GB avg = 0.025 KB total ukey len = 0.240108 GB avg = 0.017 KB total ukey num = 0.013866070 Billion total entry num = 0.013866070 Billion write speed all = 169.046716198 MB/sec (with seq num) write speed all = 165.271380509 MB/sec (without seq num) 2024/11/19-15:19:06.396837 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4702: 32059 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:06.396890 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000746396868, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4702, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 115783218, "largest_seqno": 115843228, "table_properties": {"data_size": 32602112, "index_size": 921984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32602092, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000746, "oldest_key_time": 0, "file_creation_time": 1732000746, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4702, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:06.426676 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-00 2024/11/19-15:19:06.426773 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00111/att-00/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:19:06.498261 139851127064320 [compaction_job.cc:1948] [default] [JOB 105] Generated table #4687: 463425 keys, 18758025 bytes, temperature: kUnknown 2024/11/19-15:19:06.498315 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000746498290, "cf_name": "default", "job": 105, "event": "table_file_creation", "file_number": 4687, "file_size": 18758025, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 17330528, "index_size": 876928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12975900, "raw_average_key_size": 28, "raw_value_size": 54133066, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463425, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000745, "oldest_key_time": 0, "file_creation_time": 1732000745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4687, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:06.508448 139851127064320 [compaction_job.cc:703] job-00105: subcompact[0], size: 0.018758 G, files: /storage/mytopling/.rocksdb [004687] 2024/11/19-15:19:06.535158 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777215 2024/11/19-15:19:06.535494 139851127064320 (Original Log Time 2024/11/19-15:19:05.155329) EVENT_LOG_v1 {"time_micros": 1732000745155311, "job": 105, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4589], "score": 0.886859, "input_data_size": 55501528, "oldest_snapshot_seqno": 121750395} 2024/11/19-15:19:06.535506 139851127064320 (Original Log Time 2024/11/19-15:19:06.534874) [compaction_job.cc:2022] [default] [JOB 105] Compacted 1@2 files to L3 => 18758025 bytes 2024/11/19-15:19:06.535511 139851127064320 (Original Log Time 2024/11/19-15:19:06.535116) [compaction_job.cc:1224] [default] compacted to: files[6 19 39 5 0 0 0] max score 0.84 (25 files need compaction), MB/sec: 41.0 rd, 13.9 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(52.9, 0.0 +0.0 blob) out(17.9 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 463425, records dropped: 0 output_compression: Snappy 2024/11/19-15:19:06.535531 139851127064320 (Original Log Time 2024/11/19-15:19:06.535148) EVENT_LOG_v1 {"time_micros": 1732000746535131, "cf": "default", "job": 105, "event": "compaction_finished", "compaction_time_micros": 1353131, "compaction_time_cpu_micros": 925075, "output_level": 3, "num_output_files": 1, "total_output_size": 18758025, "num_input_records": 463425, "num_output_records": 463425, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 19, 39, 5, 0, 0, 0]} 2024/11/19-15:19:06.535624 139851127064320 [compaction_job.cc:2437] [default] [JOB 114] Compacting 1@2 files to L3, score 0.84, subcompactions 1 : 1 2024/11/19-15:19:06.535635 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 112 Base level 2, inputs: [4598(52MB)] 2024/11/19-15:19:06.703078 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4703: 32131 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:06.703122 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000746703101, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4703, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 115810463, "largest_seqno": 115871609, "table_properties": {"data_size": 32676672, "index_size": 847360, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642620, "raw_average_key_size": 20, "raw_value_size": 32676617, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32131, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000746, "oldest_key_time": 0, "file_creation_time": 1732000746, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4703, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:06.750388 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-01 2024/11/19-15:19:06.750460 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00107/att-01/compact.done) = No such file or directory: 7.049 sec on http probe failed: 3, retry 2024/11/19-15:19:06.776717 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-00 2024/11/19-15:19:06.776785 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00112/att-00/compact.done) = No such file or directory: 6.757 sec on http probe failed: 5, giveup 2024/11/19-15:19:06.776803 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00112/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 64.001 sec, real 7.083 (submit 318.330 ms wait 6.757 s), input 64.001 MiB, send shutdown 2024/11/19-15:19:06.785036 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-00 2024/11/19-15:19:06.805300 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4704: 32058 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:06.805351 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000746805327, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4704, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 115871610, "largest_seqno": 115980084, "table_properties": {"data_size": 32602176, "index_size": 921536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602130, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000746, "oldest_key_time": 0, "file_creation_time": 1732000746, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4704, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:06.840223 139851127064320 [dcompact_etcd.cc:1163] job-00114/att-00: init time wrt = 0.001, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.304 sec, input = 64.000 MiB 2024/11/19-15:19:06.907159 139851158521600 [dcompact_etcd.cc:1773] job-00112/att-00: CleanFiles 0.122 sec : 0.000 2024/11/19-15:19:07.008912 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4705: 32052 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:07.008994 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000747008962, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4705, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 115980085, "largest_seqno": 116644682, "table_properties": {"data_size": 32596608, "index_size": 926976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641040, "raw_average_key_size": 20, "raw_value_size": 32596577, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32052, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000746, "oldest_key_time": 0, "file_creation_time": 1732000746, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4705, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:07.031102 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-02 2024/11/19-15:19:07.031181 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00108/att-02/compact.done) = No such file or directory: 2.887 sec on http probe failed: 2, retry 2024/11/19-15:19:07.062051 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4706: 32150 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:07.062101 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000747062078, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4706, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 116644683, "largest_seqno": 116726113, "table_properties": {"data_size": 32696512, "index_size": 827200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32696477, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000747, "oldest_key_time": 0, "file_creation_time": 1732000747, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4706, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:07.162110 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4707: 32053 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:07.162161 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000747162140, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4707, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 116726114, "largest_seqno": 116798805, "table_properties": {"data_size": 32598080, "index_size": 925824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32598042, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000747, "oldest_key_time": 0, "file_creation_time": 1732000747, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4707, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:07.177518 139851158521600 [dcompact_etcd.cc:1163] job-00112/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.270 sec, input = 64.001 MiB 2024/11/19-15:19:07.263521 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4708: 32055 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:07.263573 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000747263550, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4708, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 116781118, "largest_seqno": 116841125, "table_properties": {"data_size": 32600256, "index_size": 923968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32600234, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000747, "oldest_key_time": 0, "file_creation_time": 1732000747, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4708, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:07.365298 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4709: 32065 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:07.365357 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000747365327, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4709, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 116790859, "largest_seqno": 116861939, "table_properties": {"data_size": 32611264, "index_size": 912896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641300, "raw_average_key_size": 20, "raw_value_size": 32611240, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32065, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000747, "oldest_key_time": 0, "file_creation_time": 1732000747, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4709, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:07.467118 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4710: 32054 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:07.467168 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000747467146, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4710, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 116861940, "largest_seqno": 116964566, "table_properties": {"data_size": 32598720, "index_size": 925376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641080, "raw_average_key_size": 20, "raw_value_size": 32598658, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000747, "oldest_key_time": 0, "file_creation_time": 1732000747, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4710, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:07.671148 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4711: 32055 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:07.671201 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000747671178, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4711, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 116964567, "largest_seqno": 117600652, "table_properties": {"data_size": 32600384, "index_size": 923520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32600348, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000747, "oldest_key_time": 0, "file_creation_time": 1732000747, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4711, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:07.874960 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4712: 32077 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:07.875017 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000747874995, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4712, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 117600653, "largest_seqno": 117676082, "table_properties": {"data_size": 32624640, "index_size": 899200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641540, "raw_average_key_size": 20, "raw_value_size": 32624625, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32077, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000747, "oldest_key_time": 0, "file_creation_time": 1732000747, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4712, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:07.992465 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4713: 32056 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:07.992525 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000747992501, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4713, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 117676083, "largest_seqno": 117788997, "table_properties": {"data_size": 32599360, "index_size": 924480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32599340, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000747, "oldest_key_time": 0, "file_creation_time": 1732000747, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4713, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.094828 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4714: 32057 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.094887 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748094864, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4714, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 117778948, "largest_seqno": 117838955, "table_properties": {"data_size": 32601792, "index_size": 922560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601730, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000747, "oldest_key_time": 0, "file_creation_time": 1732000747, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4714, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.167214 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4715: 32108 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.167271 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748167250, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4715, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 117781053, "largest_seqno": 117849173, "table_properties": {"data_size": 32651520, "index_size": 872640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642160, "raw_average_key_size": 20, "raw_value_size": 32651471, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32108, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4715, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.200398 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4716: 32054 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.200450 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748200428, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4716, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 117849174, "largest_seqno": 117923876, "table_properties": {"data_size": 32599744, "index_size": 924416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641080, "raw_average_key_size": 20, "raw_value_size": 32599741, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4716, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.247573 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4717: 32149 keys, 33524288 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.247626 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748247604, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4717, "file_size": 33524288, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 117923877, "largest_seqno": 118027051, "table_properties": {"data_size": 32696960, "index_size": 826496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642980, "raw_average_key_size": 20, "raw_value_size": 32696958, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32149, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4717, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.279395 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4718: 32060 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.279446 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748279424, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4718, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 118027052, "largest_seqno": 118666716, "table_properties": {"data_size": 32604160, "index_size": 920128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32604115, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4718, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.314816 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4719: 32065 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.314866 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748314845, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4719, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 118666717, "largest_seqno": 118769432, "table_properties": {"data_size": 32609664, "index_size": 913920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641300, "raw_average_key_size": 20, "raw_value_size": 32609642, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32065, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4719, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.316054 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-02 2024/11/19-15:19:08.316131 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00108/att-02/compact.done) = No such file or directory: 4.171 sec on http probe failed: 3, retry 2024/11/19-15:19:08.346375 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4720: 32058 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.346420 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748346400, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4720, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 118769433, "largest_seqno": 118831500, "table_properties": {"data_size": 32602368, "index_size": 921472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602319, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4720, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.381822 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4721: 32056 keys, 33524800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.381872 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748381851, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4721, "file_size": 33524800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 118776894, "largest_seqno": 118836901, "table_properties": {"data_size": 32603968, "index_size": 920000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32603957, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4721, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.417069 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4722: 32084 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.417115 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748417096, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4722, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 118823555, "largest_seqno": 118911755, "table_properties": {"data_size": 32628864, "index_size": 895552, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641680, "raw_average_key_size": 20, "raw_value_size": 32628860, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32084, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4722, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.436342 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-00 2024/11/19-15:19:08.436440 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00111/att-00/compact.done) = No such file or directory: 9.017 sec on http probe failed: 3, retry 2024/11/19-15:19:08.448494 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-00 2024/11/19-15:19:08.448581 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00114/att-00/compact.done) = No such file or directory: 1.600 sec on http probe failed: 1, retry 2024/11/19-15:19:08.448775 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4723: 32058 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.448819 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748448800, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4723, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 118911756, "largest_seqno": 119021204, "table_properties": {"data_size": 32602368, "index_size": 921344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602308, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4723, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.482305 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4724: 32058 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.482362 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748482341, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4724, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119021205, "largest_seqno": 119654704, "table_properties": {"data_size": 32602624, "index_size": 921024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602604, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4724, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.515558 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4725: 32118 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.515607 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748515587, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4725, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119654705, "largest_seqno": 119729440, "table_properties": {"data_size": 32662464, "index_size": 861376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642360, "raw_average_key_size": 20, "raw_value_size": 32662443, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32118, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4725, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.550811 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4726: 32059 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.550859 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748550839, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4726, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119729441, "largest_seqno": 119812595, "table_properties": {"data_size": 32603328, "index_size": 921152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603315, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4726, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.582252 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4727: 32059 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.582297 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748582278, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4727, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119775559, "largest_seqno": 119835566, "table_properties": {"data_size": 32603648, "index_size": 920768, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603625, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4727, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.615487 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4728: 32149 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.615535 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748615515, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4728, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119804653, "largest_seqno": 119896869, "table_properties": {"data_size": 32696896, "index_size": 826816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642980, "raw_average_key_size": 20, "raw_value_size": 32696891, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32149, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4728, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.648674 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4729: 32056 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.648720 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748648700, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4729, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119896870, "largest_seqno": 119974638, "table_properties": {"data_size": 32601792, "index_size": 922688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32601730, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4729, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.681804 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4730: 32055 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.681852 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748681832, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4730, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119974639, "largest_seqno": 120638942, "table_properties": {"data_size": 32599104, "index_size": 924672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32599045, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4730, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.716719 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4731: 32149 keys, 33524288 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.716765 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748716745, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4731, "file_size": 33524288, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120638943, "largest_seqno": 120717015, "table_properties": {"data_size": 32697088, "index_size": 826368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642980, "raw_average_key_size": 20, "raw_value_size": 32697080, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32149, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4731, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.747970 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4732: 32055 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.748016 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748747996, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4732, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120717016, "largest_seqno": 120793036, "table_properties": {"data_size": 32600768, "index_size": 923456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32600751, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4732, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.759796 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-01 2024/11/19-15:19:08.759889 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00107/att-01/compact.done) = No such file or directory: 9.058 sec on http probe failed: 4, retry 2024/11/19-15:19:08.783027 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4733: 32056 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.783069 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748783049, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4733, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120773480, "largest_seqno": 120833487, "table_properties": {"data_size": 32601152, "index_size": 922880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32601149, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4733, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.787082 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-01 2024/11/19-15:19:08.787152 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00112/att-01/compact.done) = No such file or directory: 1.600 sec on http probe failed: 1, retry 2024/11/19-15:19:08.813996 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4734: 32131 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.814037 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748814017, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4734, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120795094, "largest_seqno": 120856237, "table_properties": {"data_size": 32677248, "index_size": 846656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642620, "raw_average_key_size": 20, "raw_value_size": 32677229, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32131, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4734, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.849041 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4735: 32058 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.849085 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748849065, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4735, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120856238, "largest_seqno": 120924950, "table_properties": {"data_size": 32602112, "index_size": 921920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602094, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4735, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.880216 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4736: 32052 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.880259 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748880239, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4736, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120924951, "largest_seqno": 121634621, "table_properties": {"data_size": 32596800, "index_size": 926912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641040, "raw_average_key_size": 20, "raw_value_size": 32596771, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32052, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4736, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.915277 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4737: 32152 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.915328 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748915300, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4737, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121634622, "largest_seqno": 121739498, "table_properties": {"data_size": 32696512, "index_size": 827200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643040, "raw_average_key_size": 20, "raw_value_size": 32696466, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32152, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4737, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.946896 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4738: 32053 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.946951 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748946929, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4738, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121739499, "largest_seqno": 121792462, "table_properties": {"data_size": 32598720, "index_size": 925696, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32598678, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4738, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:08.965400 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-02 2024/11/19-15:19:08.965488 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00102/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:19:08.978621 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4739: 32057 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:08.978663 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000748978643, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4739, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121760401, "largest_seqno": 121840411, "table_properties": {"data_size": 32600576, "index_size": 923776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32600567, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4739, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:09.010649 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4740: 32071 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:09.010695 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000749010675, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4740, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121764515, "largest_seqno": 121826595, "table_properties": {"data_size": 32616256, "index_size": 907968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641420, "raw_average_key_size": 20, "raw_value_size": 32616224, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32071, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000748, "oldest_key_time": 0, "file_creation_time": 1732000748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4740, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:09.033854 139849728263936 [compaction_job.cc:1948] [default] [JOB 113] Generated table #4741: 23815 keys, 24869696 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:09.033898 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000749033878, "cf_name": "default", "job": 113, "event": "table_file_creation", "file_number": 4741, "file_size": 24869696, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121826596, "largest_seqno": 121860417, "table_properties": {"data_size": 24220608, "index_size": 648256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 476300, "raw_average_key_size": 20, "raw_value_size": 24220578, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 23815, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000749, "oldest_key_time": 0, "file_creation_time": 1732000749, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4741, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:19:09.033996 139851137550080 [compaction_job.cc:703] job-00113: subcompact[0], size: 0.327269 G, files: /storage/mytopling/.rocksdb [004634,004635,004636,004637,004638,004640,004642,004644,004647,004648,004650,004651,004653] 2024/11/19-15:19:09.034018 139851137550080 [compaction_job.cc:703] job-00113: subcompact[1], size: 0.370105 G, files: /storage/mytopling/.rocksdb [004632,004641,004643,004645,004652,004655,004657,004659,004661,004663,004665,004667,004670,004673] 2024/11/19-15:19:09.034049 139851137550080 [compaction_job.cc:703] job-00113: subcompact[2], size: 2.719649 G, files: /storage/mytopling/.rocksdb [004633,004639,004646,004649,004654,004656,004658,004660,004662,004664,004666,004668,004669,004671,004672,004674,004675,004676,004677,004678,004679,004680,004681,004682,004683,004684,004685,004686,004688,004689,004690,004691,004692,004693,004694,004695,004696,004697,004698,004699,004700,004701,004702,004703,004704,004705,004706,004707,004708,004709,004710,004711,004712,004713,004714,004715,004716,004717,004718,004719,004720,004721,004722,004723,004724,004725,004726,004727,004728,004729,004730,004731,004732,004733,004734,004735,004736,004737,004738,004739,004740,004741] 2024/11/19-15:19:09.128254 139851137550080 (Original Log Time 2024/11/19-15:19:02.902834) EVENT_LOG_v1 {"time_micros": 1732000742902824, "job": 113, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [4627, 4605, 4596, 4586, 4579, 4574], "files_L1": [4548, 4549, 4550, 4551, 4553, 4555, 4556, 4557, 4558, 4559, 4560, 4561, 4562, 4565, 4566, 4569, 4570, 4571, 4572], "score": 5.27786, "input_data_size": 3853386289, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:09.128266 139851137550080 (Original Log Time 2024/11/19-15:19:09.127494) [compaction_job.cc:2022] [default] [JOB 113] Compacted 6@0 + 19@1 files to L1 => 3417022524 bytes 2024/11/19-15:19:09.128269 139851137550080 (Original Log Time 2024/11/19-15:19:09.127958) [compaction_job.cc:1224] [default] compacted to: files[0 109 39 5 0 0 0] max score 6.37 (133 files need compaction), MB/sec: 628.5 rd, 557.3 wr, level 1, files in(6, 19) out(109 +0 blob) MB in(3089.5, 585.4 +0.0 blob) out(3258.7 +0.0 blob), read-write-amplify(2.2) write-amplify(1.1) OK, records in: 8623909, records dropped: 0 output_compression: Snappy 2024/11/19-15:19:09.128279 139851137550080 (Original Log Time 2024/11/19-15:19:09.128011) EVENT_LOG_v1 {"time_micros": 1732000749127970, "cf": "default", "job": 113, "event": "compaction_finished", "compaction_time_micros": 6131218, "compaction_time_cpu_micros": 5238216, "output_level": 1, "num_output_files": 109, "total_output_size": 3417022524, "num_input_records": 8623909, "num_output_records": 8623909, "num_subcompactions": 3, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 109, 39, 5, 0, 0, 0]} 2024/11/19-15:19:09.128356 139851137550080 [compaction_job.cc:2437] [default] [JOB 115] Compacting 2@1 files to L2, score 6.37, subcompactions 1 : 1 2024/11/19-15:19:09.128365 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 113 Base level 1, inputs: [4634(6131B) 4635(27KB)] 2024/11/19-15:19:09.425905 139851137550080 [dcompact_etcd.cc:1163] job-00115/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.297 sec, input = 24.350 KiB 2024/11/19-15:19:09.496808 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-00 2024/11/19-15:19:09.496867 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00115/att-00/compact.done) = No such file or directory: 0.062 sec on http probe failed: 1, retry 2024/11/19-15:19:09.505172 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-00 2024/11/19-15:19:09.505243 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00115/att-00/compact.done) = No such file or directory: 0.071 sec on http probe failed: 2, retry 2024/11/19-15:19:09.514430 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-00 2024/11/19-15:19:09.514492 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00115/att-00/compact.done) = No such file or directory: 0.080 sec on http probe failed: 3, retry 2024/11/19-15:19:09.523870 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-00 2024/11/19-15:19:09.523930 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00115/att-00/compact.done) = No such file or directory: 0.089 sec on http probe failed: 4, retry 2024/11/19-15:19:09.528869 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-00 2024/11/19-15:19:09.528917 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00115/att-00/compact.done) = No such file or directory: 0.099 sec on http probe failed: 5, giveup 2024/11/19-15:19:09.528939 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00115/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.400 (submit 297.006 ms wait 0.099 s), input 24.350 KiB, send shutdown 2024/11/19-15:19:09.536290 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-00 2024/11/19-15:19:09.536509 139851137550080 [dcompact_etcd.cc:1773] job-00115/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:09.600737 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-02 2024/11/19-15:19:09.600791 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00108/att-02/compact.done) = No such file or directory: 5.456 sec on http probe failed: 4, retry 2024/11/19-15:19:09.733591 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-00 2024/11/19-15:19:09.733711 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00114/att-00/compact.done) = No such file or directory: 2.888 sec on http probe failed: 2, retry 2024/11/19-15:19:09.808828 139851137550080 [dcompact_etcd.cc:1163] job-00115/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.272 sec, input = 24.350 KiB 2024/11/19-15:19:09.888488 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-01 2024/11/19-15:19:09.888569 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00115/att-01/compact.done) = No such file or directory: 0.072 sec on http probe failed: 1, retry 2024/11/19-15:19:09.896615 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-01 2024/11/19-15:19:09.896678 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00115/att-01/compact.done) = No such file or directory: 0.080 sec on http probe failed: 2, retry 2024/11/19-15:19:09.905872 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-01 2024/11/19-15:19:09.905942 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00115/att-01/compact.done) = No such file or directory: 0.088 sec on http probe failed: 3, retry 2024/11/19-15:19:09.911860 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-01 2024/11/19-15:19:09.911940 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00115/att-01/compact.done) = No such file or directory: 0.098 sec on http probe failed: 4, retry 2024/11/19-15:19:09.920733 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-01 2024/11/19-15:19:09.920806 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00115/att-01/compact.done) = No such file or directory: 0.104 sec on http probe failed: 5, giveup 2024/11/19-15:19:09.920837 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00115/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.384 (submit 272.194 ms wait 0.104 s), input 24.350 KiB, send shutdown 2024/11/19-15:19:09.929776 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-01 2024/11/19-15:19:09.930038 139851137550080 [dcompact_etcd.cc:1773] job-00115/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:10.071844 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-01 2024/11/19-15:19:10.071954 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00112/att-01/compact.done) = No such file or directory: 2.890 sec on http probe failed: 2, retry 2024/11/19-15:19:10.242321 139851137550080 [dcompact_etcd.cc:1163] job-00115/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.312 sec, input = 24.350 KiB 2024/11/19-15:19:10.307483 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-02 2024/11/19-15:19:10.307548 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00115/att-02/compact.done) = No such file or directory: 0.060 sec on http probe failed: 1, retry 2024/11/19-15:19:10.312599 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-02 2024/11/19-15:19:10.312646 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00115/att-02/compact.done) = No such file or directory: 0.066 sec on http probe failed: 2, retry 2024/11/19-15:19:10.322599 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-02 2024/11/19-15:19:10.322641 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00115/att-02/compact.done) = No such file or directory: 0.071 sec on http probe failed: 3, retry 2024/11/19-15:19:10.332569 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-02 2024/11/19-15:19:10.332614 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00115/att-02/compact.done) = No such file or directory: 0.081 sec on http probe failed: 4, retry 2024/11/19-15:19:10.341323 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-02 2024/11/19-15:19:10.341370 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00115/att-02/compact.done) = No such file or directory: 0.091 sec on http probe failed: 5, giveup 2024/11/19-15:19:10.341384 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00115/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.411 (submit 312.109 ms wait 0.091 s), input 24.350 KiB, send shutdown 2024/11/19-15:19:10.349645 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00115/att-02 2024/11/19-15:19:10.349872 139851137550080 [dcompact_etcd.cc:1773] job-00115/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:10.351657 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00115) = success 2024/11/19-15:19:10.351664 139851137550080 [dcompact_etcd.cc:1773] job-00115/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:19:10.351681 139851137550080 (Original Log Time 2024/11/19-15:19:09.128378) EVENT_LOG_v1 {"time_micros": 1732000749128372, "job": 115, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4634, 4635], "score": 6.37149, "input_data_size": 34155, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:10.351689 139851137550080 [compaction_job.cc:2437] [default] [JOB 115] Compacting 2@1 files to L2, score 6.37, subcompactions 1 : 1 2024/11/19-15:19:10.351696 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 113 Base level 1, inputs: [4634(6131B) 4635(27KB)] 2024/11/19-15:19:10.351978 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004742.sst) /storage/mytopling/..//tmp/Topling-82038-hxuY86 2024/11/19-15:19:10.352090 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004742.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:19:10.352175 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004742.sst): index pass time = 0.00's, 1.025'MB/sec index type = UintIndex_AllOne usrkeys = 16 min-keylen = 4 max-keylen = 4 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 10.00 2024/11/19-15:19:10.352210 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004742.sst): first pass time = 0.00's, 9.808'MB/sec 2024/11/19-15:19:10.352237 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004742.sst): old prealloc_size = 37570, real_size = 2528 2024/11/19-15:19:10.352252 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004742.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 16 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 10.00 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 120.75 avg-zval = 135.00 2024/11/19-15:19:10.352462 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004742.sst): second pass time = 0.00's,29723.077'MB/sec, value only(88.3% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 17.039'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 11.967'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.142'MB/sec write SST data time = 0.00's, 21.441'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 29723.077'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 16 avg-key = 16.00 avg-zkey = 12.00 avg-val = 120.75 avg-zval = 135.00 usrkeys = 16 avg-key = 8.00 avg-zkey = 12.00 avg-val = 127.75 avg-zval = 135.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 16 bytes = 64 } seq expand size = 112 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.6667 value = 0.8944 dict = 0.00 all = 0.6100 } Zip/UnZip{ index = 1.5000 value = 1.1180 dict = 0.00 all = 1.6394 } ---------------------------- total value len = 4.615970 GB avg = 0.333 KB (by entry num) total key len = 0.351037 GB avg = 0.025 KB total ukey len = 0.240108 GB avg = 0.017 KB total ukey num = 0.013866086 Billion total entry num = 0.013866086 Billion write speed all = 148.782702736 MB/sec (with seq num) write speed all = 145.459922808 MB/sec (without seq num) 2024/11/19-15:19:10.352647 139851137550080 [compaction_job.cc:1948] [default] [JOB 115] Generated table #4742: 16 keys, 3587 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:10.352680 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000750352660, "cf_name": "default", "job": 115, "event": "table_file_creation", "file_number": 4742, "file_size": 3587, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 114603865, "largest_seqno": 121760399, "table_properties": {"data_size": 2160, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 256, "raw_average_key_size": 16, "raw_value_size": 1932, "raw_average_value_size": 120, "num_data_blocks": 1, "num_entries": 16, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000750, "oldest_key_time": 0, "file_creation_time": 1732000750, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4742, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:10.352845 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004743.sst) /storage/mytopling/..//tmp/Topling-82038-WaHp2l 2024/11/19-15:19:10.352992 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004743.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:19:10.353119 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004743.sst): first pass time = 0.00's, 85.062'MB/sec 2024/11/19-15:19:10.353142 139834035275520 [fixed_len_key_index.cc:363] FixedLenKeyIndex build_cache Finish: num_keys = 161, fixlen = 8, pref_len = 4, cplen = 3, time = 0.000 sec 2024/11/19-15:19:10.353203 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004743.sst): index pass time = 0.00's, 6.661'MB/sec index type = FixedLenKeyIndex usrkeys = 161 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 60.42 2024/11/19-15:19:10.353241 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004743.sst): old prealloc_size = 37570, real_size = 29984 2024/11/19-15:19:10.353256 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004743.sst): index type = FixedLenKeyIndex , store type = PlainBlobStore usrkeys = 161 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 60.42 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 121.28 avg-zval = 124.52 2024/11/19-15:19:10.353474 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004743.sst): second pass time = 0.00's,207723.404'MB/sec, value only(85.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 142.006'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 614.685'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 1.167'MB/sec write SST data time = 0.00's, 182.693'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 207723.404'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 161 avg-key = 20.00 avg-zkey = 60.42 avg-val = 121.28 avg-zval = 124.52 usrkeys = 161 avg-key = 12.00 avg-zkey = 60.42 avg-val = 128.28 avg-zval = 124.52 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 161 bytes = 480 } seq expand size = 1127 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.1986 value = 0.9740 dict = 0.00 all = 0.7235 } Zip/UnZip{ index = 5.0352 value = 1.0267 dict = 0.00 all = 1.3822 } ---------------------------- total value len = 4.615990 GB avg = 0.333 KB (by entry num) total key len = 0.351040 GB avg = 0.025 KB total ukey len = 0.240110 GB avg = 0.017 KB total ukey num = 0.013866247 Billion total entry num = 0.013866247 Billion write speed all = 148.778902364 MB/sec (with seq num) write speed all = 145.456184700 MB/sec (without seq num) 2024/11/19-15:19:10.353816 139851137550080 [compaction_job.cc:1948] [default] [JOB 115] Generated table #4743: 161 keys, 31440 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:10.353845 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000750353829, "cf_name": "default", "job": 115, "event": "table_file_creation", "file_number": 4743, "file_size": 31440, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 113825375, "largest_seqno": 121850416, "table_properties": {"data_size": 20048, "index_size": 9728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3220, "raw_average_key_size": 20, "raw_value_size": 19526, "raw_average_value_size": 121, "num_data_blocks": 1, "num_entries": 161, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-FixedLenKeyIndex-PlainBlobStore", "creation_time": 1732000750, "oldest_key_time": 0, "file_creation_time": 1732000750, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4743, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:10.353941 139851137550080 [compaction_job.cc:703] job-00115: subcompact[0], size: 0.000035 G, files: /storage/mytopling/.rocksdb [004742,004743] 2024/11/19-15:19:10.354529 139851137550080 (Original Log Time 2024/11/19-15:19:10.351723) EVENT_LOG_v1 {"time_micros": 1732000750351708, "job": 115, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4634, 4635], "score": 6.37149, "input_data_size": 34155, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:10.354537 139851137550080 (Original Log Time 2024/11/19-15:19:10.354203) [compaction_job.cc:2022] [default] [JOB 115] Compacted 2@1 files to L2 => 35027 bytes 2024/11/19-15:19:10.354541 139851137550080 (Original Log Time 2024/11/19-15:19:10.354441) [compaction_job.cc:1224] [default] compacted to: files[0 107 41 5 0 0 0] max score 6.37 (133 files need compaction), MB/sec: 15.4 rd, 15.7 wr, level 2, files in(2, 0) out(2 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 177, records dropped: 0 output_compression: Snappy 2024/11/19-15:19:10.354552 139851137550080 (Original Log Time 2024/11/19-15:19:10.354459) EVENT_LOG_v1 {"time_micros": 1732000750354450, "cf": "default", "job": 115, "event": "compaction_finished", "compaction_time_micros": 2224, "compaction_time_cpu_micros": 1767, "output_level": 2, "num_output_files": 2, "total_output_size": 35027, "num_input_records": 177, "num_output_records": 177, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 107, 41, 5, 0, 0, 0]} 2024/11/19-15:19:10.354612 139851137550080 [compaction_job.cc:2437] [default] [JOB 116] Compacting 4@1 files to L2, score 6.37, subcompactions 1 : 1 2024/11/19-15:19:10.354620 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 114 Base level 1, inputs: [4636(31MB) 4637(31MB) 4638(31MB) 4640(31MB)] 2024/11/19-15:19:10.442184 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-00 2024/11/19-15:19:10.442260 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00111/att-00/compact.done) = No such file or directory: 11.027 sec on http probe failed: 4, retry 2024/11/19-15:19:10.623157 139851137550080 [dcompact_etcd.cc:1163] job-00116/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.268 sec, input = 127.238 MiB 2024/11/19-15:19:10.768063 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-01 2024/11/19-15:19:10.768152 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00107/att-01/compact.done) = No such file or directory: 11.068 sec on http probe failed: 5, giveup 2024/11/19-15:19:10.768172 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00107/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 121.222 sec, real 11.383 (submit 307.138 ms wait 11.068 s), input 121.222 MiB, send shutdown 2024/11/19-15:19:10.773300 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-01 2024/11/19-15:19:10.773537 139850978166528 [dcompact_etcd.cc:1773] job-00107/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:10.882381 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-02 2024/11/19-15:19:10.882459 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00108/att-02/compact.done) = No such file or directory: 6.741 sec on http probe failed: 5, giveup 2024/11/19-15:19:10.882476 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00108/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 63.818 sec, real 7.035 (submit 289.046 ms wait 6.741 s), input 63.818 MiB, send shutdown 2024/11/19-15:19:10.887819 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00108/att-02 2024/11/19-15:19:10.888054 139851169007360 [dcompact_etcd.cc:1773] job-00108/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:10.888298 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00108) = success 2024/11/19-15:19:10.888304 139851169007360 [dcompact_etcd.cc:1773] job-00108/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:10.888322 139851169007360 (Original Log Time 2024/11/19-15:18:49.727230) EVENT_LOG_v1 {"time_micros": 1732000729727222, "job": 108, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4592], "score": 1.01406, "input_data_size": 55356184, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:10.888329 139851169007360 [compaction_job.cc:2437] [default] [JOB 108] Compacting 1@2 files to L3, score 1.01, subcompactions 1 : 1 2024/11/19-15:19:10.888337 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 106 Base level 2, inputs: [4592(52MB)] 2024/11/19-15:19:10.888567 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004744.sst) /storage/mytopling/..//tmp/Topling-82038-HUdg0e 2024/11/19-15:19:10.973650 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-02 2024/11/19-15:19:10.973736 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00102/att-02/compact.done) = No such file or directory: 7.008 sec on http probe failed: 2, retry 2024/11/19-15:19:11.019459 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-00 2024/11/19-15:19:11.019537 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00114/att-00/compact.done) = No such file or directory: 4.174 sec on http probe failed: 3, retry 2024/11/19-15:19:11.051652 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004744.sst): first pass time = 0.16's, 410.357'MB/sec 2024/11/19-15:19:11.051654 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004744.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:19:11.051677 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004744.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.006 GB, dictZip workingMem = 0.0033 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:19:11.066707 139850978166528 [dcompact_etcd.cc:1163] job-00107/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.293 sec, input = 121.222 MiB 2024/11/19-15:19:11.086086 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004744.sst): index pass time = 0.03's, 214.930'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462188 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:19:11.370985 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-01 2024/11/19-15:19:11.371097 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00112/att-01/compact.done) = No such file or directory: 4.175 sec on http probe failed: 3, retry 2024/11/19-15:19:11.693228 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004744.sst): old prealloc_size = 60891802, real_size = 18324400 2024/11/19-15:19:11.693496 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004744.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 462188 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0173 GB avg-val = 116.79 avg-zval = 37.49 2024/11/19-15:19:11.693566 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004744.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.642 sec, Key+Value = 0.067 GB 2024/11/19-15:19:11.954101 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004744.sst): second pass time = 0.64's, 84.139'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.26's, 71.911'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3430.446'MB/sec (index lex order gen) rebuild zvType time = 0.21's, 0.541'MB/sec write SST data time = 0.05's, 399.192'MB/sec dict compress time = 0.00's, 1598.452'MB/sec z-dict build time = 0.00's, sample length = 0.544'MB, throughput = 8104.507'MB/sec zip my value time = 0.64's, unzip length = 0.054'GB zip my value throughput = 84.139'MB/sec zip pipeline throughput = 84.139'MB/sec entries = 462188 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 37.49 usrkeys = 462188 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 37.49 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 10 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.54 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0173 GB dict = 0.54 MB all = 0.0188 GB } UnZip/Zip{ index = 10.5012 value = 3.1149 dict = 1.00 all = 3.5674 } Zip/UnZip{ index = 0.0952 value = 0.3210 dict = 1.00 all = 0.2803 } ---------------------------- total value len = 4.669967 GB avg = 0.326 KB (by entry num) total key len = 0.363982 GB avg = 0.025 KB total ukey len = 0.249354 GB avg = 0.017 KB total ukey num = 0.014328435 Billion total entry num = 0.014328435 Billion write speed all = 143.884962917 MB/sec (with seq num) write speed all = 140.608574630 MB/sec (without seq num) 2024/11/19-15:19:11.955059 139851169007360 [compaction_job.cc:1948] [default] [JOB 108] Generated table #4744: 462188 keys, 18758207 bytes, temperature: kUnknown 2024/11/19-15:19:11.955111 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000751955088, "cf_name": "default", "job": 108, "event": "table_file_creation", "file_number": 4744, "file_size": 18758207, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 17328416, "index_size": 880256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12941264, "raw_average_key_size": 28, "raw_value_size": 53976956, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462188, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000750, "oldest_key_time": 0, "file_creation_time": 1732000750, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4744, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:11.965199 139851169007360 [compaction_job.cc:703] job-00108: subcompact[0], size: 0.018758 G, files: /storage/mytopling/.rocksdb [004744] 2024/11/19-15:19:11.966193 139851169007360 (Original Log Time 2024/11/19-15:19:10.888363) EVENT_LOG_v1 {"time_micros": 1732000750888346, "job": 108, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4592], "score": 1.01406, "input_data_size": 55356184, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:11.966203 139851169007360 (Original Log Time 2024/11/19-15:19:11.965831) [compaction_job.cc:2022] [default] [JOB 108] Compacted 1@2 files to L3 => 18758207 bytes 2024/11/19-15:19:11.966206 139851169007360 (Original Log Time 2024/11/19-15:19:11.966097) [compaction_job.cc:1224] [default] compacted to: files[0 107 40 6 0 0 0] max score 6.12 (129 files need compaction), MB/sec: 51.4 rd, 17.4 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(52.8, 0.0 +0.0 blob) out(17.9 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 462188, records dropped: 0 output_compression: Snappy 2024/11/19-15:19:11.966218 139851169007360 (Original Log Time 2024/11/19-15:19:11.966118) EVENT_LOG_v1 {"time_micros": 1732000751966107, "cf": "default", "job": 108, "event": "compaction_finished", "compaction_time_micros": 1076849, "compaction_time_cpu_micros": 879492, "output_level": 3, "num_output_files": 1, "total_output_size": 18758207, "num_input_records": 462188, "num_output_records": 462188, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 107, 40, 6, 0, 0, 0]} 2024/11/19-15:19:11.966291 139851169007360 [compaction_job.cc:2437] [default] [JOB 117] Compacting 4@1 files to L2, score 6.12, subcompactions 1 : 1 2024/11/19-15:19:11.966300 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 115 Base level 1, inputs: [4642(31MB) 4644(31MB) 4647(31MB) 4648(31MB)] 2024/11/19-15:19:12.276401 139851169007360 [dcompact_etcd.cc:1163] job-00117/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.310 sec, input = 127.241 MiB 2024/11/19-15:19:12.307836 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-00 2024/11/19-15:19:12.307933 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00114/att-00/compact.done) = No such file or directory: 5.459 sec on http probe failed: 4, retry 2024/11/19-15:19:12.450466 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-00 2024/11/19-15:19:12.450535 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00111/att-00/compact.done) = No such file or directory: 13.033 sec on http probe failed: 5, giveup 2024/11/19-15:19:12.450555 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00111/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 241.761 sec, real 13.374 (submit 332.881 ms wait 13.033 s), input 241.761 MiB, send shutdown 2024/11/19-15:19:12.456058 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-00 2024/11/19-15:19:12.456310 139851179493120 [dcompact_etcd.cc:1773] job-00111/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:12.659593 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-01 2024/11/19-15:19:12.659676 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00112/att-01/compact.done) = No such file or directory: 5.474 sec on http probe failed: 4, retry 2024/11/19-15:19:12.734647 139851179493120 [dcompact_etcd.cc:1163] job-00111/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.278 sec, input = 241.761 MiB 2024/11/19-15:19:12.982782 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-02 2024/11/19-15:19:12.982869 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00102/att-02/compact.done) = No such file or directory: 9.017 sec on http probe failed: 3, retry 2024/11/19-15:19:13.596512 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-00 2024/11/19-15:19:13.596613 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00114/att-00/compact.done) = No such file or directory: 6.748 sec on http probe failed: 5, giveup 2024/11/19-15:19:13.596633 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00114/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 64.000 sec, real 7.060 (submit 303.882 ms wait 6.748 s), input 64.000 MiB, send shutdown 2024/11/19-15:19:13.605759 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-00 2024/11/19-15:19:13.606039 139851127064320 [dcompact_etcd.cc:1773] job-00114/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:13.812566 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-00 2024/11/19-15:19:13.812675 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00116/att-00/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:19:13.867297 139851127064320 [dcompact_etcd.cc:1163] job-00114/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.261 sec, input = 64.000 MiB 2024/11/19-15:19:13.948884 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-01 2024/11/19-15:19:13.948980 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00112/att-01/compact.done) = No such file or directory: 6.762 sec on http probe failed: 5, giveup 2024/11/19-15:19:13.948999 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00112/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 64.001 sec, real 7.042 (submit 270.226 ms wait 6.762 s), input 64.001 MiB, send shutdown 2024/11/19-15:19:13.957436 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-01 2024/11/19-15:19:13.957671 139851158521600 [dcompact_etcd.cc:1773] job-00112/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:14.106489 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-02 2024/11/19-15:19:14.106587 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00107/att-02/compact.done) = No such file or directory: 3.031 sec on http probe failed: 1, retry 2024/11/19-15:19:14.239079 139851158521600 [dcompact_etcd.cc:1163] job-00112/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 64.001 MiB 2024/11/19-15:19:14.923978 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4745. Immutable memtables: 0. 2024/11/19-15:19:14.924263 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 118] Syncing log #4626 2024/11/19-15:19:14.924357 139850957195008 (Original Log Time 2024/11/19-15:19:14.924053) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:19:14.924365 139850957195008 [flush_job.cc:879] [default] [JOB 118] Flushing memtable with next log file: 4745 2024/11/19-15:19:14.924386 139850957195008 EVENT_LOG_v1 {"time_micros": 1732000754924381, "job": 118, "event": "flush_started", "num_memtables": 1, "num_entries": 1472133, "num_deletes": 0, "total_data_size": 458508901, "memory_usage": 540016008, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:19:14.924390 139850957195008 [flush_job.cc:910] [default] [JOB 118] Level-0 flush table #4746: started 2024/11/19-15:19:14.924702 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000012.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004746.sst in ConvertToSST, mem_size = 514.999 M, time = 303.185 us 2024/11/19-15:19:14.991674 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-02 2024/11/19-15:19:14.991809 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00102/att-02/compact.done) = No such file or directory: 11.026 sec on http probe failed: 4, retry 2024/11/19-15:19:15.144951 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004746.sst): fsize = 515.001 M, time(ms): open: 0.034, seek: 0.015, finish: 0.038, meta: 0.013, Flush: 0.013, sync: 220.110, close: 0.010, all: 220.233 2024/11/19-15:19:15.146023 139850957195008 [flush_job.cc:1083] [default] [JOB 118] Flush lasted 221665 microseconds, and 177376 cpu microseconds. 2024/11/19-15:19:15.166589 139850957195008 (Original Log Time 2024/11/19-15:19:15.145003) [flush_job.cc:1029] [default] [JOB 118] Level-0 flush table #4746: 540017434 bytes OK (needs compaction) 2024/11/19-15:19:15.166600 139850957195008 (Original Log Time 2024/11/19-15:19:15.146039) [memtable_list.cc:560] [default] Level-0 commit table #4746 started 2024/11/19-15:19:15.166604 139850957195008 (Original Log Time 2024/11/19-15:19:15.166515) [memtable_list.cc:759] [default] Level-0 commit table #4746: memtable #1 done 2024/11/19-15:19:15.166607 139850957195008 (Original Log Time 2024/11/19-15:19:15.166541) EVENT_LOG_v1 {"time_micros": 1732000755166535, "job": 118, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [1, 107, 40, 6, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:19:15.166610 139850957195008 (Original Log Time 2024/11/19-15:19:15.166561) [db_impl_compaction_flush.cc:310] [default] Level summary: files[1 107 40 6 0 0 0] max score 5.87 (126 files need compaction) 2024/11/19-15:19:15.467342 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-00 2024/11/19-15:19:15.467446 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00117/att-00/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:19:15.472730 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-01 2024/11/19-15:19:15.472827 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00114/att-01/compact.done) = No such file or directory: 1.600 sec on http probe failed: 1, retry 2024/11/19-15:19:15.821895 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-00 2024/11/19-15:19:15.821981 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00116/att-00/compact.done) = No such file or directory: 5.190 sec on http probe failed: 2, retry 2024/11/19-15:19:15.847073 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-02 2024/11/19-15:19:15.847147 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00112/att-02/compact.done) = No such file or directory: 1.600 sec on http probe failed: 1, retry 2024/11/19-15:19:16.115719 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-02 2024/11/19-15:19:16.115796 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00107/att-02/compact.done) = No such file or directory: 5.040 sec on http probe failed: 2, retry 2024/11/19-15:19:16.758019 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-01 2024/11/19-15:19:16.758117 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00114/att-01/compact.done) = No such file or directory: 2.886 sec on http probe failed: 2, retry 2024/11/19-15:19:16.999852 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-02 2024/11/19-15:19:16.999935 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00102/att-02/compact.done) = No such file or directory: 13.035 sec on http probe failed: 5, giveup 2024/11/19-15:19:16.999954 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00102/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 254.856 sec, real 13.328 (submit 285.555 ms wait 13.035 s), input 254.856 MiB, send shutdown 2024/11/19-15:19:17.007662 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00102/att-02 2024/11/19-15:19:17.007896 139851148035840 [dcompact_etcd.cc:1773] job-00102/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:17.008154 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00102) = success 2024/11/19-15:19:17.008166 139851148035840 [dcompact_etcd.cc:1773] job-00102/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:17.008187 139851148035840 (Original Log Time 2024/11/19-15:18:37.013321) EVENT_LOG_v1 {"time_micros": 1732000717013313, "job": 102, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4577, 4580, 4582, 4587], "score": 1.02431, "input_data_size": 259632756, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:17.008194 139851148035840 [compaction_job.cc:2437] [default] [JOB 102] Compacting 4@2 files to L3, score 1.02, subcompactions 1 : 1 2024/11/19-15:19:17.008202 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 100 Base level 2, inputs: [4577(61MB) 4580(62MB) 4582(61MB) 4587(62MB)] 2024/11/19-15:19:17.008463 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004747.sst) /storage/mytopling/..//tmp/Topling-82038-pLe25W 2024/11/19-15:19:17.137509 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-02 2024/11/19-15:19:17.137612 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00112/att-02/compact.done) = No such file or directory: 2.888 sec on http probe failed: 2, retry 2024/11/19-15:19:17.140771 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004747.sst): first pass time = 0.13's,1014.533'MB/sec 2024/11/19-15:19:17.140775 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004747.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0018 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:19:17.140798 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004747.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.002 GB, dictZip workingMem = 0.0081 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:19:17.151460 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004747.sst): index pass time = 0.01's, 225.921'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 200669 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 2024/11/19-15:19:17.478386 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-00 2024/11/19-15:19:17.478509 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00117/att-00/compact.done) = No such file or directory: 5.193 sec on http probe failed: 2, retry 2024/11/19-15:19:17.745555 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-01 2024/11/19-15:19:17.745664 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00111/att-01/compact.done) = No such file or directory: 5.001 sec on http probe failed: 1, retry 2024/11/19-15:19:17.843301 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-00 2024/11/19-15:19:17.843412 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00116/att-00/compact.done) = No such file or directory: 7.201 sec on http probe failed: 3, retry 2024/11/19-15:19:18.055764 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-01 2024/11/19-15:19:18.055878 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00114/att-01/compact.done) = No such file or directory: 4.175 sec on http probe failed: 3, retry 2024/11/19-15:19:18.131549 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-02 2024/11/19-15:19:18.131648 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00107/att-02/compact.done) = No such file or directory: 7.050 sec on http probe failed: 3, retry 2024/11/19-15:19:18.439987 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-02 2024/11/19-15:19:18.440087 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00112/att-02/compact.done) = No such file or directory: 4.179 sec on http probe failed: 3, retry 2024/11/19-15:19:19.340701 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-01 2024/11/19-15:19:19.340783 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00114/att-01/compact.done) = No such file or directory: 5.469 sec on http probe failed: 4, retry 2024/11/19-15:19:19.374293 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004747.sst): old prealloc_size = 147639500, real_size = 103771536 2024/11/19-15:19:19.374427 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004747.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 200669 prefix = 4 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 raw-val = 0.1294 GB zip-val = 0.1034 GB avg-val = 644.85 avg-zval = 515.19 2024/11/19-15:19:19.374478 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004747.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0005 GB, waited 2.234 sec, Key+Value = 0.134 GB 2024/11/19-15:19:19.487523 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-00 2024/11/19-15:19:19.487611 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00117/att-00/compact.done) = No such file or directory: 7.202 sec on http probe failed: 3, retry 2024/11/19-15:19:19.580933 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004747.sst): second pass time = 2.23's, 57.938'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.21's, 508.546'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2887.292'MB/sec (index lex order gen) rebuild zvType time = 0.16's, 0.307'MB/sec write SST data time = 0.04's, 2439.548'MB/sec dict compress time = 0.00's, 1348.897'MB/sec z-dict build time = 0.00's, sample length = 1.346'MB, throughput = 13149.685'MB/sec zip my value time = 2.23's, unzip length = 0.129'GB zip my value throughput = 57.938'MB/sec zip pipeline throughput = 57.938'MB/sec entries = 200669 avg-key = 24.00 avg-zkey = 1.69 avg-val = 644.85 avg-zval = 515.19 usrkeys = 200669 avg-key = 16.00 avg-zkey = 1.69 avg-val = 644.85 avg-zval = 515.19 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0032 GB value = 0.1294 GB dict = 1.35 MB all = 0.1342 GB } __ZipSize{ index = 0.0003 GB value = 0.1034 GB dict = 1.35 MB all = 0.1051 GB } UnZip/Zip{ index = 9.4798 value = 1.2517 dict = 1.00 all = 1.2774 } Zip/UnZip{ index = 0.1055 value = 0.7989 dict = 1.00 all = 0.7828 } ---------------------------- total value len = 4.799369 GB avg = 0.330 KB (by entry num) total key len = 0.368798 GB avg = 0.025 KB total ukey len = 0.252565 GB avg = 0.017 KB total ukey num = 0.014529104 Billion total entry num = 0.014529104 Billion write speed all = 121.282172401 MB/sec (with seq num) write speed all = 118.554518479 MB/sec (without seq num) 2024/11/19-15:19:19.674894 139851148035840 [compaction_job.cc:1948] [default] [JOB 102] Generated table #4747: 200669 keys, 105070615 bytes, temperature: kUnknown 2024/11/19-15:19:19.674960 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000759674935, "cf_name": "default", "job": 102, "event": "table_file_creation", "file_number": 4747, "file_size": 105070615, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 103382528, "index_size": 338688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4816056, "raw_average_key_size": 24, "raw_value_size": 129401971, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 200669, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000757, "oldest_key_time": 0, "file_creation_time": 1732000757, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4747, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:19.704829 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004748.sst) /storage/mytopling/..//tmp/Topling-82038-L8dfpE 2024/11/19-15:19:19.728839 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-02 2024/11/19-15:19:19.728917 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00112/att-02/compact.done) = No such file or directory: 5.481 sec on http probe failed: 4, retry 2024/11/19-15:19:19.753978 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-01 2024/11/19-15:19:19.754041 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00111/att-01/compact.done) = No such file or directory: 7.011 sec on http probe failed: 2, retry 2024/11/19-15:19:19.848538 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-00 2024/11/19-15:19:19.848730 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00116/att-00/compact.done) = No such file or directory: 9.220 sec on http probe failed: 4, retry 2024/11/19-15:19:19.868956 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004748.sst): first pass time = 0.16's, 810.571'MB/sec 2024/11/19-15:19:19.868961 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004748.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0018 GB, waited 0.000 sec, Key+Value = 0.133 GB 2024/11/19-15:19:19.868984 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004748.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.002 GB, dictZip workingMem = 0.0080 GB, waited 0.000 sec, Key+Value = 0.133 GB 2024/11/19-15:19:19.884726 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004748.sst): index pass time = 0.02's, 151.600'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 198819 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 2024/11/19-15:19:20.136820 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-02 2024/11/19-15:19:20.136899 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00107/att-02/compact.done) = No such file or directory: 9.065 sec on http probe failed: 4, retry 2024/11/19-15:19:20.638241 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-01 2024/11/19-15:19:20.638341 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00114/att-01/compact.done) = No such file or directory: 6.755 sec on http probe failed: 5, giveup 2024/11/19-15:19:20.638366 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00114/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 64.000 sec, real 7.032 (submit 261.087 ms wait 6.755 s), input 64.000 MiB, send shutdown 2024/11/19-15:19:20.654631 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-01 2024/11/19-15:19:20.654910 139851127064320 [dcompact_etcd.cc:1773] job-00114/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:20.932887 139851127064320 [dcompact_etcd.cc:1163] job-00114/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.278 sec, input = 64.000 MiB 2024/11/19-15:19:21.025852 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-02 2024/11/19-15:19:21.025968 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00112/att-02/compact.done) = No such file or directory: 6.772 sec on http probe failed: 5, giveup 2024/11/19-15:19:21.025994 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00112/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 64.001 sec, real 7.068 (submit 281.332 ms wait 6.772 s), input 64.001 MiB, send shutdown 2024/11/19-15:19:21.036529 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00112/att-02 2024/11/19-15:19:21.036798 139851158521600 [dcompact_etcd.cc:1773] job-00112/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:21.037095 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00112) = success 2024/11/19-15:19:21.037107 139851158521600 [dcompact_etcd.cc:1773] job-00112/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:21.037129 139851158521600 (Original Log Time 2024/11/19-15:18:59.693472) EVENT_LOG_v1 {"time_micros": 1732000739693466, "job": 112, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4622], "score": 0.856456, "input_data_size": 66137796, "oldest_snapshot_seqno": 122477319} 2024/11/19-15:19:21.037140 139851158521600 [compaction_job.cc:2437] [default] [JOB 112] Compacting 1@2 files to L3, score 0.86, subcompactions 1 : 1 2024/11/19-15:19:21.037150 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 110 Base level 2, inputs: [4622(63MB)] 2024/11/19-15:19:21.037474 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004749.sst) /storage/mytopling/..//tmp/Topling-82038-tV6ygz 2024/11/19-15:19:21.152366 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004749.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.008 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:19:21.158080 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004749.sst): index pass time = 0.01's, 91.230'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64716 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:19:21.158145 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004749.sst): first pass time = 0.12's, 556.204'MB/sec 2024/11/19-15:19:21.158164 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004749.sst: sumWaitingMem = 0.004 GB, sumWorkingMem = 0.008 GB, dictZip workingMem = 0.0040 GB, waited 0.006 sec, Key+Value = 0.067 GB 2024/11/19-15:19:21.503432 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-00 2024/11/19-15:19:21.503533 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00117/att-00/compact.done) = No such file or directory: 9.212 sec on http probe failed: 4, retry 2024/11/19-15:19:21.765702 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-01 2024/11/19-15:19:21.765815 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00111/att-01/compact.done) = No such file or directory: 9.020 sec on http probe failed: 3, retry 2024/11/19-15:19:21.863110 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-00 2024/11/19-15:19:21.863204 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00116/att-00/compact.done) = No such file or directory: 11.229 sec on http probe failed: 5, giveup 2024/11/19-15:19:21.863229 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00116/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.238 sec, real 11.508 (submit 268.075 ms wait 11.229 s), input 127.238 MiB, send shutdown 2024/11/19-15:19:21.874091 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-00 2024/11/19-15:19:21.875964 139851137550080 [dcompact_etcd.cc:1773] job-00116/att-00: CleanFiles 0.002 sec : 0.000 2024/11/19-15:19:22.150456 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-02 2024/11/19-15:19:22.150569 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00107/att-02/compact.done) = No such file or directory: 11.071 sec on http probe failed: 5, giveup 2024/11/19-15:19:22.150593 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00107/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 121.222 sec, real 11.377 (submit 293.071 ms wait 11.071 s), input 121.222 MiB, send shutdown 2024/11/19-15:19:22.165255 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00107/att-02 2024/11/19-15:19:22.165576 139850978166528 [dcompact_etcd.cc:1773] job-00107/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:22.165913 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00107) = success 2024/11/19-15:19:22.165938 139850978166528 [dcompact_etcd.cc:1773] job-00107/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:22.165964 139850978166528 (Original Log Time 2024/11/19-15:18:47.987798) EVENT_LOG_v1 {"time_micros": 1732000727987790, "job": 107, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4581, 4583, 4584, 4591], "score": 1.01033, "input_data_size": 93953993, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:22.165975 139850978166528 [compaction_job.cc:2437] [default] [JOB 107] Compacting 4@2 files to L3, score 1.01, subcompactions 1 : 1 2024/11/19-15:19:22.165987 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 105 Base level 2, inputs: [4581(26MB) 4583(240KB) 4584(10MB) 4591(52MB)] 2024/11/19-15:19:22.166383 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004750.sst) /storage/mytopling/..//tmp/Topling-82038-tSzJYJ 2024/11/19-15:19:22.188076 139851137550080 [dcompact_etcd.cc:1163] job-00116/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.312 sec, input = 127.238 MiB 2024/11/19-15:19:22.370622 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004748.sst): old prealloc_size = 147639500, real_size = 102855168 2024/11/19-15:19:22.370877 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004748.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 198819 prefix = 4 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 raw-val = 0.1282 GB zip-val = 0.1025 GB avg-val = 645.04 avg-zval = 515.39 2024/11/19-15:19:22.370998 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004748.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.004 GB, reorder workingMem = 0.0004 GB, waited 2.502 sec, Key+Value = 0.133 GB 2024/11/19-15:19:22.457127 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004750.sst): first pass time = 0.29's, 122.184'MB/sec 2024/11/19-15:19:22.457129 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004750.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.004 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.036 GB 2024/11/19-15:19:22.457167 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004750.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.004 GB, dictZip workingMem = 0.0016 GB, waited 0.000 sec, Key+Value = 0.036 GB 2024/11/19-15:19:22.457262 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004750.sst): index pass time = 0.00's,40288.736'MB/sec index type = UintIndex_AllOne usrkeys = 480000 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0038 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:19:22.480347 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004749.sst): old prealloc_size = 72751575, real_size = 20253616 2024/11/19-15:19:22.480436 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004749.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 64716 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0201 GB avg-val =1016.99 avg-zval = 311.01 2024/11/19-15:19:22.480537 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004749.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.002 GB, reorder workingMem = 0.0001 GB, waited 1.328 sec, Key+Value = 0.067 GB 2024/11/19-15:19:22.541099 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-02 2024/11/19-15:19:22.541200 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00114/att-02/compact.done) = No such file or directory: 1.600 sec on http probe failed: 1, retry 2024/11/19-15:19:22.543822 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004749.sst): second pass time = 1.32's, 49.779'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 329.504'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1607.562'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.344'MB/sec write SST data time = 0.02's, 1279.671'MB/sec dict compress time = 0.00's, 1291.573'MB/sec z-dict build time = 0.00's, sample length = 0.673'MB, throughput = 6742.070'MB/sec zip my value time = 1.32's, unzip length = 0.066'GB zip my value throughput = 49.779'MB/sec zip pipeline throughput = 81.227'MB/sec entries = 64716 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 311.01 usrkeys = 64716 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 311.01 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.67 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0201 GB dict = 0.67 MB all = 0.0209 GB } UnZip/Zip{ index = 7.0466 value = 3.2700 dict = 1.00 all = 3.2091 } Zip/UnZip{ index = 0.1419 value = 0.3058 dict = 1.00 all = 0.3116 } ---------------------------- total value len = 4.865185 GB avg = 0.333 KB (by entry num) total key len = 0.370092 GB avg = 0.025 KB total ukey len = 0.253341 GB avg = 0.017 KB total ukey num = 0.014593820 Billion total entry num = 0.014593820 Billion write speed all = 114.870027671 MB/sec (with seq num) write speed all = 112.308340641 MB/sec (without seq num) 2024/11/19-15:19:22.545097 139851158521600 [compaction_job.cc:1948] [default] [JOB 112] Generated table #4749: 64716 keys, 20912294 bytes, temperature: kUnknown 2024/11/19-15:19:22.545147 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000762545122, "cf_name": "default", "job": 112, "event": "table_file_creation", "file_number": 4749, "file_size": 20912294, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 20127072, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294320, "raw_average_key_size": 20, "raw_value_size": 65815465, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64716, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000761, "oldest_key_time": 0, "file_creation_time": 1732000761, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4749, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:22.555672 139851158521600 [compaction_job.cc:703] job-00112: subcompact[0], size: 0.020912 G, files: /storage/mytopling/.rocksdb [004749] 2024/11/19-15:19:22.556627 139851158521600 (Original Log Time 2024/11/19-15:19:21.037188) EVENT_LOG_v1 {"time_micros": 1732000761037166, "job": 112, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4622], "score": 0.856456, "input_data_size": 66137796, "oldest_snapshot_seqno": 122477319} 2024/11/19-15:19:22.556637 139851158521600 (Original Log Time 2024/11/19-15:19:22.556231) [compaction_job.cc:2022] [default] [JOB 112] Compacted 1@2 files to L3 => 20912294 bytes 2024/11/19-15:19:22.556640 139851158521600 (Original Log Time 2024/11/19-15:19:22.556502) [compaction_job.cc:1224] [default] compacted to: files[1 107 39 7 0 0 0] max score 5.87 (126 files need compaction), MB/sec: 43.6 rd, 13.8 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(63.1, 0.0 +0.0 blob) out(19.9 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 64716, records dropped: 0 output_compression: Snappy 2024/11/19-15:19:22.556653 139851158521600 (Original Log Time 2024/11/19-15:19:22.556528) EVENT_LOG_v1 {"time_micros": 1732000762556515, "cf": "default", "job": 112, "event": "compaction_finished", "compaction_time_micros": 1518497, "compaction_time_cpu_micros": 469691, "output_level": 3, "num_output_files": 1, "total_output_size": 20912294, "num_input_records": 64716, "num_output_records": 64716, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 107, 39, 7, 0, 0, 0]} 2024/11/19-15:19:22.556720 139851158521600 [compaction_job.cc:2437] [default] [JOB 119] Compacting 2@1 files to L2, score 5.87, subcompactions 1 : 1 2024/11/19-15:19:22.556728 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 117 Base level 1, inputs: [4650(31MB) 4651(22MB)] 2024/11/19-15:19:22.645039 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004748.sst): second pass time = 2.50's, 51.266'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.27's, 379.568'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1464.719'MB/sec (index lex order gen) rebuild zvType time = 0.20's, 0.244'MB/sec write SST data time = 0.07's, 1484.222'MB/sec dict compress time = 0.00's, 2001.288'MB/sec z-dict build time = 0.00's, sample length = 1.341'MB, throughput = 20792.506'MB/sec zip my value time = 2.50's, unzip length = 0.128'GB zip my value throughput = 51.266'MB/sec zip pipeline throughput = 60.565'MB/sec entries = 198819 avg-key = 24.00 avg-zkey = 1.69 avg-val = 645.04 avg-zval = 515.39 usrkeys = 198819 avg-key = 16.00 avg-zkey = 1.69 avg-val = 645.04 avg-zval = 515.39 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0032 GB value = 0.1282 GB dict = 1.34 MB all = 0.1330 GB } __ZipSize{ index = 0.0003 GB value = 0.1025 GB dict = 1.34 MB all = 0.1041 GB } UnZip/Zip{ index = 9.4766 value = 1.2515 dict = 1.00 all = 1.2772 } Zip/UnZip{ index = 0.1055 value = 0.7990 dict = 1.00 all = 0.7830 } ---------------------------- total value len = 4.993430 GB avg = 0.338 KB (by entry num) total key len = 0.374864 GB avg = 0.025 KB total ukey len = 0.256523 GB avg = 0.017 KB total ukey num = 0.014792639 Billion total entry num = 0.014792639 Billion write speed all = 117.527675809 MB/sec (with seq num) write speed all = 114.936842350 MB/sec (without seq num) 2024/11/19-15:19:22.651971 139851148035840 [compaction_job.cc:1948] [default] [JOB 102] Generated table #4748: 198819 keys, 104149364 bytes, temperature: kUnknown 2024/11/19-15:19:22.652043 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000762652008, "cf_name": "default", "job": 102, "event": "table_file_creation", "file_number": 4748, "file_size": 104149364, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 102469616, "index_size": 335680, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4771656, "raw_average_key_size": 24, "raw_value_size": 128245837, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 198819, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000759, "oldest_key_time": 0, "file_creation_time": 1732000759, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4748, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:22.709274 139851148035840 [compaction_job.cc:703] job-00102: subcompact[0], size: 0.209220 G, files: /storage/mytopling/.rocksdb [004747,004748] 2024/11/19-15:19:22.711889 139851148035840 (Original Log Time 2024/11/19-15:19:17.008234) EVENT_LOG_v1 {"time_micros": 1732000757008214, "job": 102, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4577, 4580, 4582, 4587], "score": 1.02431, "input_data_size": 259632756, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:22.711905 139851148035840 (Original Log Time 2024/11/19-15:19:22.711334) [compaction_job.cc:2022] [default] [JOB 102] Compacted 4@2 files to L3 => 209219979 bytes 2024/11/19-15:19:22.711911 139851148035840 (Original Log Time 2024/11/19-15:19:22.711714) [compaction_job.cc:1224] [default] compacted to: files[1 107 35 9 0 0 0] max score 5.77 (124 files need compaction), MB/sec: 45.5 rd, 36.7 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(247.6, 0.0 +0.0 blob) out(199.5 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 399488, records dropped: 0 output_compression: Snappy 2024/11/19-15:19:22.712049 139851148035840 (Original Log Time 2024/11/19-15:19:22.711751) EVENT_LOG_v1 {"time_micros": 1732000762711733, "cf": "default", "job": 102, "event": "compaction_finished", "compaction_time_micros": 5701064, "compaction_time_cpu_micros": 1655597, "output_level": 3, "num_output_files": 2, "total_output_size": 209219979, "num_input_records": 399488, "num_output_records": 399488, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 107, 35, 9, 0, 0, 0]} 2024/11/19-15:19:22.721951 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004536.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:22.721984 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000762721980, "job": 102, "event": "table_file_deletion", "file_number": 4536} 2024/11/19-15:19:22.731946 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004537.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:22.731980 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000762731976, "job": 102, "event": "table_file_deletion", "file_number": 4537} 2024/11/19-15:19:22.745945 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004538.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:22.745978 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000762745974, "job": 102, "event": "table_file_deletion", "file_number": 4538} 2024/11/19-15:19:22.756495 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004539.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:22.756525 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000762756522, "job": 102, "event": "table_file_deletion", "file_number": 4539} 2024/11/19-15:19:22.769945 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004540.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:22.769975 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000762769971, "job": 102, "event": "table_file_deletion", "file_number": 4540} 2024/11/19-15:19:22.784944 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004541.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:22.784974 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000762784971, "job": 102, "event": "table_file_deletion", "file_number": 4541} 2024/11/19-15:19:22.793415 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004542.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:22.793441 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000762793439, "job": 102, "event": "table_file_deletion", "file_number": 4542} 2024/11/19-15:19:22.797907 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004543.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:22.797935 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000762797933, "job": 102, "event": "table_file_deletion", "file_number": 4543} 2024/11/19-15:19:22.802243 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004552.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:22.802262 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000762802260, "job": 102, "event": "table_file_deletion", "file_number": 4552} 2024/11/19-15:19:22.806116 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004564.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:22.806134 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000762806132, "job": 102, "event": "table_file_deletion", "file_number": 4564} 2024/11/19-15:19:22.810419 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004567.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:22.810438 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000762810436, "job": 102, "event": "table_file_deletion", "file_number": 4567} 2024/11/19-15:19:22.810593 139851148035840 [compaction_job.cc:2437] [default] [JOB 120] Compacting 3@1 files to L2, score 5.77, subcompactions 1 : 1 2024/11/19-15:19:22.810602 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 118 Base level 1, inputs: [4653(2514KB) 4632(31MB) 4641(2077KB)] 2024/11/19-15:19:22.874002 139851158521600 [dcompact_etcd.cc:1163] job-00119/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.317 sec, input = 53.697 MiB 2024/11/19-15:19:22.981249 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004750.sst): old prealloc_size = 103349392, real_size = 15944952 2024/11/19-15:19:22.981292 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004750.sst): index type = UintIndex_AllOne , store type = DictZipBlobStore usrkeys = 480000 prefix = 4 raw-key = 0.0038 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0259 GB zip-val = 0.0158 GB avg-val = 54.00 avg-zval = 32.97 2024/11/19-15:19:23.018589 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004750.sst): second pass time = 0.52's, 49.464'MB/sec, value only(73.0% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.04's, 430.971'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 8.178'MB/sec (index lex order gen) rebuild zvType time = 0.01's, 14.109'MB/sec write SST data time = 0.03's, 558.606'MB/sec dict compress time = 0.00's, 1176.774'MB/sec z-dict build time = 0.00's, sample length = 0.259'MB, throughput = 3094.580'MB/sec zip my value time = 0.52's, unzip length = 0.026'GB zip my value throughput = 49.464'MB/sec zip pipeline throughput = 49.464'MB/sec entries = 480000 avg-key = 20.00 avg-zkey = 0.00 avg-val = 54.00 avg-zval = 32.97 usrkeys = 480000 avg-key = 12.00 avg-zkey = 0.00 avg-val = 54.00 avg-zval = 32.97 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 3 UnZipSize{ index = 0.0058 GB value = 0.0259 GB dict = 0.26 MB all = 0.0355 GB } __ZipSize{ index = 0.0000 GB value = 0.0158 GB dict = 0.26 MB all = 0.0161 GB } UnZip/Zip{ index =30000.0000 value = 1.6381 dict = 1.00 all = 2.2084 } Zip/UnZip{ index = 0.0000 value = 0.6105 dict = 1.00 all = 0.4528 } ---------------------------- total value len = 5.019352 GB avg = 0.329 KB (by entry num) total key len = 0.384464 GB avg = 0.025 KB total ukey len = 0.262283 GB avg = 0.017 KB total ukey num = 0.015272639 Billion total entry num = 0.015272639 Billion write speed all = 117.345647360 MB/sec (with seq num) write speed all = 114.692444100 MB/sec (without seq num) 2024/11/19-15:19:23.019389 139850978166528 [compaction_job.cc:1948] [default] [JOB 107] Generated table #4750: 480000 keys, 16085076 bytes, temperature: kUnknown 2024/11/19-15:19:23.019441 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000763019416, "cf_name": "default", "job": 107, "event": "table_file_creation", "file_number": 4750, "file_size": 16085076, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 15824576, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 9600000, "raw_average_key_size": 20, "raw_value_size": 25921743, "raw_average_value_size": 54, "num_data_blocks": 1, "num_entries": 480000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-DictZipBlobStore", "creation_time": 1732000762, "oldest_key_time": 0, "file_creation_time": 1732000762, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4750, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:23.025518 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004751.sst) /storage/mytopling/..//tmp/Topling-82038-YOuo2d 2024/11/19-15:19:23.063233 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004751.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0013 GB, waited 0.000 sec, Key+Value = 0.003 GB 2024/11/19-15:19:23.064943 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004751.sst): first pass time = 0.04's, 87.694'MB/sec 2024/11/19-15:19:23.070688 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004751.sst): index pass time = 0.01's, 232.495'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 144000 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0017 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:19:23.070746 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004751.sst): old prealloc_size = 103349392, real_size = 280416 2024/11/19-15:19:23.070830 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004751.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZeroLengthBlobStore usrkeys = 144000 prefix = 4 raw-key = 0.0017 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 0.00 avg-zval = 0.00 2024/11/19-15:19:23.070884 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004751.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.008 sec, Key+Value = 0.003 GB 2024/11/19-15:19:23.082877 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004751.sst): second pass time = 0.00's, 0.000'MB/sec, value only( 0.0% of KV) wait indexing time = 0.01's, remap KeyValue time = 0.01's, 20.316'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3105.887'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 175.142'MB/sec write SST data time = 0.01's, 20.802'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 0.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 144000 avg-key = 24.00 avg-zkey = 1.70 avg-val = 0.00 avg-zval = 0.00 usrkeys = 144000 avg-key = 16.00 avg-zkey = 1.70 avg-val = 0.00 avg-zval = 0.00 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 5 UnZipSize{ index = 0.0023 GB value = 0.0000 GB dict = 0.00 MB all = 0.0035 GB } __ZipSize{ index = 0.0002 GB value = 0.0000 GB dict = 0.00 MB all = 0.0002 GB } UnZip/Zip{ index = 9.4364 value = 0.0000 dict = 0.00 all = 14.0318 } Zip/UnZip{ index = 0.1060 value = inf dict = 0.00 all = 0.0713 } ---------------------------- total value len = 5.019352 GB avg = 0.326 KB (by entry num) total key len = 0.387920 GB avg = 0.025 KB total ukey len = 0.264587 GB avg = 0.017 KB total ukey num = 0.015416639 Billion total entry num = 0.015416639 Billion write speed all = 117.256971843 MB/sec (with seq num) write speed all = 114.582486855 MB/sec (without seq num) 2024/11/19-15:19:23.083204 139850978166528 [compaction_job.cc:1948] [default] [JOB 107] Generated table #4751: 144000 keys, 246297 bytes, temperature: kUnknown 2024/11/19-15:19:23.083246 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000763083225, "cf_name": "default", "job": 107, "event": "table_file_creation", "file_number": 4751, "file_size": 246297, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 80, "index_size": 244160, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3456000, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 144000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZeroLengthBlobStore", "creation_time": 1732000763, "oldest_key_time": 0, "file_creation_time": 1732000763, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4751, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:23.083818 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004752.sst) /storage/mytopling/..//tmp/Topling-82038-PhLJyD 2024/11/19-15:19:23.086248 139851148035840 [dcompact_etcd.cc:1163] job-00120/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.275 sec, input = 33.874 MiB 2024/11/19-15:19:23.218536 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004752.sst: sumWaitingMem = 0.004 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0043 GB, waited 0.000 sec, Key+Value = 0.021 GB 2024/11/19-15:19:23.243297 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004752.sst): index pass time = 0.02's, 232.958'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 480000 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0058 GB zip-key = 0.0008 GB avg-key = 12.00 avg-zkey = 1.68 2024/11/19-15:19:23.255616 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004752.sst): first pass time = 0.17's, 122.385'MB/sec 2024/11/19-15:19:23.255661 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004752.sst): old prealloc_size = 103349392, real_size = 10469584 2024/11/19-15:19:23.255872 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004752.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 480000 prefix = 4 raw-key = 0.0058 GB zip-key = 0.0008 GB avg-key = 12.00 avg-zkey = 1.68 raw-val = 0.0095 GB zip-val = 0.0095 GB avg-val = 19.80 avg-zval = 19.88 2024/11/19-15:19:23.255960 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004752.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.037 sec, Key+Value = 0.021 GB 2024/11/19-15:19:23.341229 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004752.sst): second pass time = 0.00's,29153374.233'MB/sec, value only(45.2% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 121.034'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3873.972'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.717'MB/sec write SST data time = 0.04's, 251.487'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.010'GB zip my value throughput = 29153374.233'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 480000 avg-key = 24.00 avg-zkey = 1.68 avg-val = 19.80 avg-zval = 19.88 usrkeys = 480000 avg-key = 16.00 avg-zkey = 1.68 avg-val = 19.80 avg-zval = 19.88 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 1166 UnZipSize{ index = 0.0077 GB value = 0.0095 GB dict = 0.00 MB all = 0.0210 GB } __ZipSize{ index = 0.0008 GB value = 0.0095 GB dict = 0.00 MB all = 0.0104 GB } UnZip/Zip{ index = 9.5042 value = 0.9961 dict = 0.00 all = 2.0304 } Zip/UnZip{ index = 0.1052 value = 1.0039 dict = 0.00 all = 0.4925 } ---------------------------- total value len = 5.028856 GB avg = 0.316 KB (by entry num) total key len = 0.399440 GB avg = 0.025 KB total ukey len = 0.272267 GB avg = 0.017 KB total ukey num = 0.015896639 Billion total entry num = 0.015896639 Billion write speed all = 117.057097536 MB/sec (with seq num) write speed all = 114.314705459 MB/sec (without seq num) 2024/11/19-15:19:23.342416 139850978166528 [compaction_job.cc:1948] [default] [JOB 107] Generated table #4752: 480000 keys, 10354513 bytes, temperature: kUnknown 2024/11/19-15:19:23.342469 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000763342445, "cf_name": "default", "job": 107, "event": "table_file_creation", "file_number": 4752, "file_size": 10354513, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 9541312, "index_size": 808064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 11520000, "raw_average_key_size": 24, "raw_value_size": 9504000, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 480000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732000763, "oldest_key_time": 0, "file_creation_time": 1732000763, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4752, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:23.346309 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004753.sst) /storage/mytopling/..//tmp/Topling-82038-IL5uvK 2024/11/19-15:19:23.509796 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-00 2024/11/19-15:19:23.509877 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004753.sst): first pass time = 0.16's, 410.313'MB/sec 2024/11/19-15:19:23.509883 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004753.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:19:23.509898 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00117/att-00/compact.done) = No such file or directory: 11.227 sec on http probe failed: 5, giveup 2024/11/19-15:19:23.509912 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004753.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.006 GB, dictZip workingMem = 0.0033 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:19:23.509931 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00117/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.241 sec, real 11.543 (submit 309.564 ms wait 11.227 s), input 127.241 MiB, send shutdown 2024/11/19-15:19:23.515260 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-00 2024/11/19-15:19:23.515509 139851169007360 [dcompact_etcd.cc:1773] job-00117/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:23.543742 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004753.sst): index pass time = 0.03's, 219.153'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463360 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:19:23.774992 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-01 2024/11/19-15:19:23.775108 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00111/att-01/compact.done) = No such file or directory: 11.031 sec on http probe failed: 4, retry 2024/11/19-15:19:23.837373 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-02 2024/11/19-15:19:23.837460 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00114/att-02/compact.done) = No such file or directory: 2.891 sec on http probe failed: 2, retry 2024/11/19-15:19:23.944509 139851169007360 [dcompact_etcd.cc:1163] job-00117/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.429 sec, input = 127.241 MiB 2024/11/19-15:19:23.948597 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-00 2024/11/19-15:19:23.948687 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00120/att-00/compact.done) = No such file or directory: 0.849 sec on http probe failed: 1, retry 2024/11/19-15:19:24.215980 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004753.sst): old prealloc_size = 103349392, real_size = 18322176 2024/11/19-15:19:24.216225 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004753.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 463360 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0173 GB avg-val = 116.83 avg-zval = 37.40 2024/11/19-15:19:24.216312 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004753.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.706 sec, Key+Value = 0.067 GB 2024/11/19-15:19:24.224645 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-00 2024/11/19-15:19:24.224717 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00119/att-00/compact.done) = No such file or directory: 1.343 sec on http probe failed: 1, retry 2024/11/19-15:19:24.480047 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004753.sst): second pass time = 0.71's, 76.673'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.26's, 71.037'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3763.882'MB/sec (index lex order gen) rebuild zvType time = 0.22's, 0.537'MB/sec write SST data time = 0.05's, 391.488'MB/sec dict compress time = 0.00's, 1519.891'MB/sec z-dict build time = 0.00's, sample length = 0.545'MB, throughput = 5704.791'MB/sec zip my value time = 0.71's, unzip length = 0.054'GB zip my value throughput = 76.673'MB/sec zip pipeline throughput = 76.673'MB/sec entries = 463360 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.83 avg-zval = 37.40 usrkeys = 463360 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.83 avg-zval = 37.40 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.55 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0173 GB dict = 0.55 MB all = 0.0188 GB } UnZip/Zip{ index = 10.5701 value = 3.1239 dict = 1.00 all = 3.5779 } Zip/UnZip{ index = 0.0946 value = 0.3201 dict = 1.00 all = 0.2795 } ---------------------------- total value len = 5.082991 GB avg = 0.311 KB (by entry num) total key len = 0.412414 GB avg = 0.025 KB total ukey len = 0.281534 GB avg = 0.017 KB total ukey num = 0.016359999 Billion total entry num = 0.016359999 Billion write speed all = 115.663840782 MB/sec (with seq num) write speed all = 112.909160562 MB/sec (without seq num) 2024/11/19-15:19:24.480939 139850978166528 [compaction_job.cc:1948] [default] [JOB 107] Generated table #4753: 463360 keys, 18756679 bytes, temperature: kUnknown 2024/11/19-15:19:24.480992 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000764480967, "cf_name": "default", "job": 107, "event": "table_file_creation", "file_number": 4753, "file_size": 18756679, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 17329392, "index_size": 876736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12974080, "raw_average_key_size": 28, "raw_value_size": 54134800, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463360, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000763, "oldest_key_time": 0, "file_creation_time": 1732000763, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4753, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:24.492741 139850978166528 [compaction_job.cc:703] job-00107: subcompact[0], size: 0.045443 G, files: /storage/mytopling/.rocksdb [004750,004751,004752,004753] 2024/11/19-15:19:24.494527 139850978166528 (Original Log Time 2024/11/19-15:19:22.166031) EVENT_LOG_v1 {"time_micros": 1732000762166009, "job": 107, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4581, 4583, 4584, 4591], "score": 1.01033, "input_data_size": 93953993, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:24.494537 139850978166528 (Original Log Time 2024/11/19-15:19:24.494147) [compaction_job.cc:2022] [default] [JOB 107] Compacted 4@2 files to L3 => 45442565 bytes 2024/11/19-15:19:24.494540 139850978166528 (Original Log Time 2024/11/19-15:19:24.494405) [compaction_job.cc:1224] [default] compacted to: files[1 107 31 13 0 0 0] max score 5.70 (121 files need compaction), MB/sec: 40.4 rd, 19.5 wr, level 3, files in(4, 0) out(4 +0 blob) MB in(89.6, 0.0 +0.0 blob) out(43.3 +0.0 blob), read-write-amplify(1.5) write-amplify(0.5) OK, records in: 1567360, records dropped: 0 output_compression: Snappy 2024/11/19-15:19:24.494553 139850978166528 (Original Log Time 2024/11/19-15:19:24.494430) EVENT_LOG_v1 {"time_micros": 1732000764494416, "cf": "default", "job": 107, "event": "compaction_finished", "compaction_time_micros": 2326722, "compaction_time_cpu_micros": 1841141, "output_level": 3, "num_output_files": 4, "total_output_size": 45442565, "num_input_records": 1567360, "num_output_records": 1567360, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 107, 31, 13, 0, 0, 0]} 2024/11/19-15:19:24.499301 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004544.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:24.499326 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000764499323, "job": 107, "event": "table_file_deletion", "file_number": 4544} 2024/11/19-15:19:24.504303 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004545.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:24.504325 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000764504322, "job": 107, "event": "table_file_deletion", "file_number": 4545} 2024/11/19-15:19:24.508753 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004546.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:24.508771 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000764508769, "job": 107, "event": "table_file_deletion", "file_number": 4546} 2024/11/19-15:19:24.513117 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004547.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:24.513136 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000764513134, "job": 107, "event": "table_file_deletion", "file_number": 4547} 2024/11/19-15:19:24.517357 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004554.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:24.517374 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000764517373, "job": 107, "event": "table_file_deletion", "file_number": 4554} 2024/11/19-15:19:24.521513 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004568.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:24.521531 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000764521529, "job": 107, "event": "table_file_deletion", "file_number": 4568} 2024/11/19-15:19:24.521674 139850978166528 [compaction_job.cc:2437] [default] [JOB 121] Compacting 2@1 files to L2, score 5.70, subcompactions 1 : 1 2024/11/19-15:19:24.521682 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 119 Base level 1, inputs: [4643(3675KB) 4645(20MB)] 2024/11/19-15:19:24.634582 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-00 2024/11/19-15:19:24.634674 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00120/att-00/compact.done) = No such file or directory: 1.540 sec on http probe failed: 2, retry 2024/11/19-15:19:24.838861 139850978166528 [dcompact_etcd.cc:1163] job-00121/att-00: init time wrt = 0.001, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.316 sec, input = 23.346 MiB 2024/11/19-15:19:25.127245 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-02 2024/11/19-15:19:25.127336 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00114/att-02/compact.done) = No such file or directory: 4.185 sec on http probe failed: 3, retry 2024/11/19-15:19:25.306416 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-00 2024/11/19-15:19:25.306510 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00119/att-00/compact.done) = No such file or directory: 2.425 sec on http probe failed: 2, retry 2024/11/19-15:19:25.319451 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-00 2024/11/19-15:19:25.319533 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00120/att-00/compact.done) = No such file or directory: 2.226 sec on http probe failed: 3, retry 2024/11/19-15:19:25.377171 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-01 2024/11/19-15:19:25.377277 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00116/att-01/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:19:25.430603 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-00 2024/11/19-15:19:25.430678 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00121/att-00/compact.done) = No such file or directory: 0.584 sec on http probe failed: 1, retry 2024/11/19-15:19:25.780243 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-01 2024/11/19-15:19:25.780346 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00111/att-01/compact.done) = No such file or directory: 13.041 sec on http probe failed: 5, giveup 2024/11/19-15:19:25.780366 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00111/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 241.761 sec, real 13.324 (submit 278.236 ms wait 13.041 s), input 241.761 MiB, send shutdown 2024/11/19-15:19:25.788199 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-01 2024/11/19-15:19:25.788412 139851179493120 [dcompact_etcd.cc:1773] job-00111/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:25.902537 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-00 2024/11/19-15:19:25.902599 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00121/att-00/compact.done) = No such file or directory: 1.059 sec on http probe failed: 2, retry 2024/11/19-15:19:26.006354 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-00 2024/11/19-15:19:26.006444 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00120/att-00/compact.done) = No such file or directory: 2.911 sec on http probe failed: 4, retry 2024/11/19-15:19:26.081830 139851179493120 [dcompact_etcd.cc:1163] job-00111/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.293 sec, input = 241.761 MiB 2024/11/19-15:19:26.378004 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-00 2024/11/19-15:19:26.378080 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00121/att-00/compact.done) = No such file or directory: 1.531 sec on http probe failed: 3, retry 2024/11/19-15:19:26.388623 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-00 2024/11/19-15:19:26.388682 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00119/att-00/compact.done) = No such file or directory: 3.507 sec on http probe failed: 3, retry 2024/11/19-15:19:26.415109 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-02 2024/11/19-15:19:26.415185 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00114/att-02/compact.done) = No such file or directory: 5.475 sec on http probe failed: 4, retry 2024/11/19-15:19:26.692930 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-00 2024/11/19-15:19:26.693064 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00120/att-00/compact.done) = No such file or directory: 3.598 sec on http probe failed: 5, giveup 2024/11/19-15:19:26.693092 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00120/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 33.874 sec, real 3.882 (submit 275.120 ms wait 3.598 s), input 33.874 MiB, send shutdown 2024/11/19-15:19:26.701393 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-00 2024/11/19-15:19:26.701655 139851148035840 [dcompact_etcd.cc:1773] job-00120/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:26.853053 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-00 2024/11/19-15:19:26.853127 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00121/att-00/compact.done) = No such file or directory: 2.006 sec on http probe failed: 4, retry 2024/11/19-15:19:26.988902 139851148035840 [dcompact_etcd.cc:1163] job-00120/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.287 sec, input = 33.874 MiB 2024/11/19-15:19:27.135357 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-01 2024/11/19-15:19:27.135442 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00117/att-01/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:19:27.326414 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-00 2024/11/19-15:19:27.326482 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00121/att-00/compact.done) = No such file or directory: 2.481 sec on http probe failed: 5, giveup 2024/11/19-15:19:27.326500 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00121/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 23.346 sec, real 2.804 (submit 316.489 ms wait 2.481 s), input 23.346 MiB, send shutdown 2024/11/19-15:19:27.331934 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-00 2024/11/19-15:19:27.332158 139850978166528 [dcompact_etcd.cc:1773] job-00121/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:27.385413 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-01 2024/11/19-15:19:27.385498 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00116/att-01/compact.done) = No such file or directory: 5.189 sec on http probe failed: 2, retry 2024/11/19-15:19:27.471483 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-00 2024/11/19-15:19:27.471567 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00119/att-00/compact.done) = No such file or directory: 4.589 sec on http probe failed: 4, retry 2024/11/19-15:19:27.645549 139850978166528 [dcompact_etcd.cc:1163] job-00121/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.313 sec, input = 23.346 MiB 2024/11/19-15:19:27.703548 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-02 2024/11/19-15:19:27.703635 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00114/att-02/compact.done) = No such file or directory: 6.762 sec on http probe failed: 5, giveup 2024/11/19-15:19:27.703653 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00114/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 64.000 sec, real 7.049 (submit 277.841 ms wait 6.762 s), input 64.000 MiB, send shutdown 2024/11/19-15:19:27.711836 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00114/att-02 2024/11/19-15:19:27.712067 139851127064320 [dcompact_etcd.cc:1773] job-00114/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:27.712307 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00114) = success 2024/11/19-15:19:27.712313 139851127064320 [dcompact_etcd.cc:1773] job-00114/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:27.712330 139851127064320 (Original Log Time 2024/11/19-15:19:06.535655) EVENT_LOG_v1 {"time_micros": 1732000746535646, "job": 114, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4598], "score": 0.835489, "input_data_size": 55493592, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:27.712337 139851127064320 [compaction_job.cc:2437] [default] [JOB 114] Compacting 1@2 files to L3, score 0.84, subcompactions 1 : 1 2024/11/19-15:19:27.712344 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 112 Base level 2, inputs: [4598(52MB)] 2024/11/19-15:19:27.712624 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004754.sst) /storage/mytopling/..//tmp/Topling-82038-lpgesh 2024/11/19-15:19:27.844481 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-01 2024/11/19-15:19:27.844581 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00120/att-01/compact.done) = No such file or directory: 0.847 sec on http probe failed: 1, retry 2024/11/19-15:19:27.874836 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004754.sst): first pass time = 0.16's, 413.744'MB/sec 2024/11/19-15:19:27.874842 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004754.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:19:27.874865 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004754.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.006 GB, dictZip workingMem = 0.0033 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:19:27.909161 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004754.sst): index pass time = 0.03's, 216.186'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463333 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:19:28.244741 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-01 2024/11/19-15:19:28.244867 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00121/att-01/compact.done) = No such file or directory: 0.584 sec on http probe failed: 1, retry 2024/11/19-15:19:28.508129 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004754.sst): old prealloc_size = 61042951, real_size = 18294448 2024/11/19-15:19:28.508375 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004754.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 463333 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0173 GB avg-val = 116.84 avg-zval = 37.34 2024/11/19-15:19:28.508465 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004754.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.634 sec, Key+Value = 0.067 GB 2024/11/19-15:19:28.530620 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-01 2024/11/19-15:19:28.530702 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00120/att-01/compact.done) = No such file or directory: 1.533 sec on http probe failed: 2, retry 2024/11/19-15:19:28.550756 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-00 2024/11/19-15:19:28.550836 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00119/att-00/compact.done) = No such file or directory: 5.672 sec on http probe failed: 5, giveup 2024/11/19-15:19:28.550860 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00119/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 53.697 sec, real 5.994 (submit 316.748 ms wait 5.672 s), input 53.697 MiB, send shutdown 2024/11/19-15:19:28.559146 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-00 2024/11/19-15:19:28.559420 139851158521600 [dcompact_etcd.cc:1773] job-00119/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:28.720088 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-01 2024/11/19-15:19:28.720162 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00121/att-01/compact.done) = No such file or directory: 1.066 sec on http probe failed: 2, retry 2024/11/19-15:19:28.769559 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004754.sst): second pass time = 0.63's, 85.491'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.26's, 71.645'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3807.736'MB/sec (index lex order gen) rebuild zvType time = 0.21's, 0.542'MB/sec write SST data time = 0.05's, 394.982'MB/sec dict compress time = 0.00's, 1537.525'MB/sec z-dict build time = 0.00's, sample length = 0.545'MB, throughput = 6229.829'MB/sec zip my value time = 0.63's, unzip length = 0.054'GB zip my value throughput = 85.491'MB/sec zip pipeline throughput = 85.491'MB/sec entries = 463333 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.84 avg-zval = 37.34 usrkeys = 463333 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.84 avg-zval = 37.34 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.55 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0173 GB dict = 0.55 MB all = 0.0187 GB } UnZip/Zip{ index = 10.5726 value = 3.1289 dict = 1.00 all = 3.5832 } Zip/UnZip{ index = 0.0946 value = 0.3196 dict = 1.00 all = 0.2791 } ---------------------------- total value len = 5.137127 GB avg = 0.305 KB (by entry num) total key len = 0.425387 GB avg = 0.025 KB total ukey len = 0.290800 GB avg = 0.017 KB total ukey num = 0.016823332 Billion total entry num = 0.016823332 Billion write speed all = 107.381554193 MB/sec (with seq num) write speed all = 104.783425867 MB/sec (without seq num) 2024/11/19-15:19:28.770517 139851127064320 [compaction_job.cc:1948] [default] [JOB 114] Generated table #4754: 463333 keys, 18728991 bytes, temperature: kUnknown 2024/11/19-15:19:28.770567 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000768770543, "cf_name": "default", "job": 114, "event": "table_file_creation", "file_number": 4754, "file_size": 18728991, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 17301920, "index_size": 876480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12973324, "raw_average_key_size": 28, "raw_value_size": 54135658, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463333, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000767, "oldest_key_time": 0, "file_creation_time": 1732000767, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4754, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:28.780436 139851127064320 [compaction_job.cc:703] job-00114: subcompact[0], size: 0.018729 G, files: /storage/mytopling/.rocksdb [004754] 2024/11/19-15:19:28.781415 139851127064320 (Original Log Time 2024/11/19-15:19:27.712372) EVENT_LOG_v1 {"time_micros": 1732000767712356, "job": 114, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [4598], "score": 0.835489, "input_data_size": 55493592, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:28.781425 139851127064320 (Original Log Time 2024/11/19-15:19:28.781072) [compaction_job.cc:2022] [default] [JOB 114] Compacted 1@2 files to L3 => 18728991 bytes 2024/11/19-15:19:28.781428 139851127064320 (Original Log Time 2024/11/19-15:19:28.781317) [compaction_job.cc:1224] [default] compacted to: files[1 107 30 14 0 0 0] max score 5.65 (119 files need compaction), MB/sec: 52.0 rd, 17.5 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(52.9, 0.0 +0.0 blob) out(17.9 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 463333, records dropped: 0 output_compression: Snappy 2024/11/19-15:19:28.781440 139851127064320 (Original Log Time 2024/11/19-15:19:28.781341) EVENT_LOG_v1 {"time_micros": 1732000768781328, "cf": "default", "job": 114, "event": "compaction_finished", "compaction_time_micros": 1068075, "compaction_time_cpu_micros": 869254, "output_level": 3, "num_output_files": 1, "total_output_size": 18728991, "num_input_records": 463333, "num_output_records": 463333, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 107, 30, 14, 0, 0, 0]} 2024/11/19-15:19:28.781506 139851127064320 [compaction_job.cc:2437] [default] [JOB 122] Compacting 4@1 files to L2, score 5.65, subcompactions 1 : 1 2024/11/19-15:19:28.781515 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 120 Base level 1, inputs: [4652(31MB) 4655(31MB) 4657(31MB) 4659(31MB)] 2024/11/19-15:19:28.825028 139851158521600 [dcompact_etcd.cc:1163] job-00119/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.265 sec, input = 53.697 MiB 2024/11/19-15:19:29.061946 139851127064320 [dcompact_etcd.cc:1163] job-00122/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 127.823 MiB 2024/11/19-15:19:29.143977 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-01 2024/11/19-15:19:29.144043 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00117/att-01/compact.done) = No such file or directory: 5.191 sec on http probe failed: 2, retry 2024/11/19-15:19:29.196763 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-01 2024/11/19-15:19:29.196831 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00121/att-01/compact.done) = No such file or directory: 1.542 sec on http probe failed: 3, retry 2024/11/19-15:19:29.216245 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-01 2024/11/19-15:19:29.216302 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00120/att-01/compact.done) = No such file or directory: 2.219 sec on http probe failed: 3, retry 2024/11/19-15:19:29.394294 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-01 2024/11/19-15:19:29.394364 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00116/att-01/compact.done) = No such file or directory: 7.198 sec on http probe failed: 3, retry 2024/11/19-15:19:29.668456 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-01 2024/11/19-15:19:29.668528 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00121/att-01/compact.done) = No such file or directory: 2.018 sec on http probe failed: 4, retry 2024/11/19-15:19:29.901693 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-01 2024/11/19-15:19:29.901765 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00120/att-01/compact.done) = No such file or directory: 2.905 sec on http probe failed: 4, retry 2024/11/19-15:19:30.146431 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-01 2024/11/19-15:19:30.146541 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00121/att-01/compact.done) = No such file or directory: 2.490 sec on http probe failed: 5, giveup 2024/11/19-15:19:30.146560 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00121/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 23.346 sec, real 2.814 (submit 313.265 ms wait 2.490 s), input 23.346 MiB, send shutdown 2024/11/19-15:19:30.155363 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-01 2024/11/19-15:19:30.155634 139850978166528 [dcompact_etcd.cc:1773] job-00121/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:30.172563 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-01 2024/11/19-15:19:30.172672 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00119/att-01/compact.done) = No such file or directory: 1.343 sec on http probe failed: 1, retry 2024/11/19-15:19:30.444345 139850978166528 [dcompact_etcd.cc:1163] job-00121/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 23.346 MiB 2024/11/19-15:19:30.587831 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-01 2024/11/19-15:19:30.587929 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00120/att-01/compact.done) = No such file or directory: 3.590 sec on http probe failed: 5, giveup 2024/11/19-15:19:30.587949 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00120/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 33.874 sec, real 3.886 (submit 287.129 ms wait 3.590 s), input 33.874 MiB, send shutdown 2024/11/19-15:19:30.597327 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-01 2024/11/19-15:19:30.597555 139851148035840 [dcompact_etcd.cc:1773] job-00120/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:30.878047 139851148035840 [dcompact_etcd.cc:1163] job-00120/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 33.874 MiB 2024/11/19-15:19:31.037692 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-02 2024/11/19-15:19:31.037804 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00121/att-02/compact.done) = No such file or directory: 0.584 sec on http probe failed: 1, retry 2024/11/19-15:19:31.091901 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-02 2024/11/19-15:19:31.092020 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00111/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:19:31.151692 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-01 2024/11/19-15:19:31.151758 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00117/att-01/compact.done) = No such file or directory: 7.200 sec on http probe failed: 3, retry 2024/11/19-15:19:31.254447 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-01 2024/11/19-15:19:31.254557 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00119/att-01/compact.done) = No such file or directory: 2.422 sec on http probe failed: 2, retry 2024/11/19-15:19:31.400210 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-01 2024/11/19-15:19:31.400279 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00116/att-01/compact.done) = No such file or directory: 9.206 sec on http probe failed: 4, retry 2024/11/19-15:19:31.513534 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-02 2024/11/19-15:19:31.513605 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00121/att-02/compact.done) = No such file or directory: 1.060 sec on http probe failed: 2, retry 2024/11/19-15:19:31.733962 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-02 2024/11/19-15:19:31.734053 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00120/att-02/compact.done) = No such file or directory: 0.847 sec on http probe failed: 1, retry 2024/11/19-15:19:31.990569 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-02 2024/11/19-15:19:31.990642 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00121/att-02/compact.done) = No such file or directory: 1.536 sec on http probe failed: 3, retry 2024/11/19-15:19:32.269856 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-00 2024/11/19-15:19:32.269968 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00122/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:19:32.336196 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-01 2024/11/19-15:19:32.336301 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00119/att-01/compact.done) = No such file or directory: 3.504 sec on http probe failed: 3, retry 2024/11/19-15:19:32.419281 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-02 2024/11/19-15:19:32.419357 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00120/att-02/compact.done) = No such file or directory: 1.534 sec on http probe failed: 2, retry 2024/11/19-15:19:32.462660 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-02 2024/11/19-15:19:32.462719 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00121/att-02/compact.done) = No such file or directory: 2.013 sec on http probe failed: 4, retry 2024/11/19-15:19:32.937112 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-02 2024/11/19-15:19:32.937179 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00121/att-02/compact.done) = No such file or directory: 2.485 sec on http probe failed: 5, giveup 2024/11/19-15:19:32.937197 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00121/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 23.346 sec, real 2.781 (submit 288.580 ms wait 2.485 s), input 23.346 MiB, send shutdown 2024/11/19-15:19:32.945394 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00121/att-02 2024/11/19-15:19:32.945598 139850978166528 [dcompact_etcd.cc:1773] job-00121/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:32.947375 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00121) = success 2024/11/19-15:19:32.947383 139850978166528 [dcompact_etcd.cc:1773] job-00121/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:19:32.947402 139850978166528 (Original Log Time 2024/11/19-15:19:24.521694) EVENT_LOG_v1 {"time_micros": 1732000764521687, "job": 121, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4643, 4645], "score": 5.69623, "input_data_size": 25037137, "oldest_snapshot_seqno": 123901122} 2024/11/19-15:19:32.947409 139850978166528 [compaction_job.cc:2437] [default] [JOB 121] Compacting 2@1 files to L2, score 5.70, subcompactions 1 : 1 2024/11/19-15:19:32.947417 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 119 Base level 1, inputs: [4643(3675KB) 4645(20MB)] 2024/11/19-15:19:32.947702 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004755.sst) /storage/mytopling/..//tmp/Topling-82038-coVfAI 2024/11/19-15:19:32.984197 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004755.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0013 GB, waited 0.000 sec, Key+Value = 0.003 GB 2024/11/19-15:19:32.985419 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004755.sst): first pass time = 0.04's, 91.656'MB/sec 2024/11/19-15:19:32.991597 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004755.sst): index pass time = 0.01's, 234.231'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 144000 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0017 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:19:32.991656 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004755.sst): old prealloc_size = 27540850, real_size = 280416 2024/11/19-15:19:32.991730 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004755.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZeroLengthBlobStore usrkeys = 144000 prefix = 4 raw-key = 0.0017 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 0.00 avg-zval = 0.00 2024/11/19-15:19:32.991809 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004755.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.008 sec, Key+Value = 0.003 GB 2024/11/19-15:19:33.004278 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004755.sst): second pass time = 0.00's, 0.000'MB/sec, value only( 0.0% of KV) wait indexing time = 0.01's, remap KeyValue time = 0.01's, 52.356'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3444.841'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 86.875'MB/sec write SST data time = 0.01's, 54.446'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 0.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 144000 avg-key = 24.00 avg-zkey = 1.70 avg-val = 0.00 avg-zval = 0.00 usrkeys = 144000 avg-key = 16.00 avg-zkey = 1.70 avg-val = 7.00 avg-zval = 0.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 144000 bytes = 414016 } seq expand size = 1008000 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0023 GB value = 0.0000 GB dict = 0.00 MB all = 0.0035 GB } __ZipSize{ index = 0.0002 GB value = 0.0000 GB dict = 0.00 MB all = 0.0007 GB } UnZip/Zip{ index = 9.4364 value = 0.0000 dict = 0.00 all = 5.2333 } Zip/UnZip{ index = 0.1060 value = inf dict = 0.00 all = 0.1911 } ---------------------------- total value len = 5.138135 GB avg = 0.303 KB (by entry num) total key len = 0.428843 GB avg = 0.025 KB total ukey len = 0.293104 GB avg = 0.017 KB total ukey num = 0.016967332 Billion total entry num = 0.016967332 Billion write speed all = 99.346244285 MB/sec (with seq num) write speed all = 96.923901946 MB/sec (without seq num) 2024/11/19-15:19:33.004993 139850978166528 [compaction_job.cc:1948] [default] [JOB 121] Generated table #4755: 144000 keys, 660390 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:33.005041 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000773005017, "cf_name": "default", "job": 121, "event": "table_file_creation", "file_number": 4755, "file_size": 660390, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 113894664, "largest_seqno": 121750395, "table_properties": {"data_size": 80, "index_size": 244160, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3456000, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 144000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZeroLengthBlobStore", "creation_time": 1732000772, "oldest_key_time": 0, "file_creation_time": 1732000772, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4755, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:33.005605 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004756.sst) /storage/mytopling/..//tmp/Topling-82038-fAVift 2024/11/19-15:19:33.097425 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-02 2024/11/19-15:19:33.097499 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00111/att-02/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:19:33.105169 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-02 2024/11/19-15:19:33.105231 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00120/att-02/compact.done) = No such file or directory: 2.219 sec on http probe failed: 3, retry 2024/11/19-15:19:33.130260 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004756.sst: sumWaitingMem = 0.004 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0043 GB, waited 0.000 sec, Key+Value = 0.021 GB 2024/11/19-15:19:33.155007 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004756.sst): index pass time = 0.02's, 233.034'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 480000 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0058 GB zip-key = 0.0008 GB avg-key = 12.00 avg-zkey = 1.68 2024/11/19-15:19:33.158743 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004756.sst): first pass time = 0.15's, 137.298'MB/sec 2024/11/19-15:19:33.158790 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004756.sst): old prealloc_size = 27540850, real_size = 10903568 2024/11/19-15:19:33.159014 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004756.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 480000 prefix = 4 raw-key = 0.0058 GB zip-key = 0.0008 GB avg-key = 12.00 avg-zkey = 1.68 raw-val = 0.0095 GB zip-val = 0.0100 GB avg-val = 19.80 avg-zval = 20.78 2024/11/19-15:19:33.159300 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-01 2024/11/19-15:19:33.159312 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004756.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.029 sec, Key+Value = 0.021 GB 2024/11/19-15:19:33.159449 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00117/att-01/compact.done) = No such file or directory: 9.207 sec on http probe failed: 4, retry 2024/11/19-15:19:33.216570 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004756.sst): second pass time = 0.00's,35070110.701'MB/sec, value only(45.2% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 210.661'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3628.617'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 7.239'MB/sec write SST data time = 0.04's, 297.051'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.010'GB zip my value throughput = 35070110.701'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 480000 avg-key = 24.00 avg-zkey = 1.68 avg-val = 19.80 avg-zval = 20.78 usrkeys = 480000 avg-key = 16.00 avg-zkey = 1.68 avg-val = 26.80 avg-zval = 20.78 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 480000 bytes = 1380016 } seq expand size = 3360000 multi value expand size = 0 cnt WriteAppend = 1171 UnZipSize{ index = 0.0077 GB value = 0.0095 GB dict = 0.00 MB all = 0.0210 GB } __ZipSize{ index = 0.0008 GB value = 0.0100 GB dict = 0.00 MB all = 0.0122 GB } UnZip/Zip{ index = 9.5042 value = 0.9528 dict = 0.00 all = 1.7277 } Zip/UnZip{ index = 0.1052 value = 1.0496 dict = 0.00 all = 0.5788 } ---------------------------- total value len = 5.150999 GB avg = 0.295 KB (by entry num) total key len = 0.440363 GB avg = 0.025 KB total ukey len = 0.300784 GB avg = 0.017 KB total ukey num = 0.017447332 Billion total entry num = 0.017447332 Billion write speed all = 99.404815787 MB/sec (with seq num) write speed all = 96.923346792 MB/sec (without seq num) 2024/11/19-15:19:33.217348 139850978166528 [compaction_job.cc:1948] [default] [JOB 121] Generated table #4756: 480000 keys, 12168541 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:33.217397 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000773217373, "cf_name": "default", "job": 121, "event": "table_file_creation", "file_number": 4756, "file_size": 12168541, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 113841381, "largest_seqno": 121699677, "table_properties": {"data_size": 9975296, "index_size": 808064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 11520000, "raw_average_key_size": 24, "raw_value_size": 9504000, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 480000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000773, "oldest_key_time": 0, "file_creation_time": 1732000773, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4756, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:33.221034 139850978166528 [compaction_job.cc:703] job-00121: subcompact[0], size: 0.012829 G, files: /storage/mytopling/.rocksdb [004755,004756] 2024/11/19-15:19:33.222044 139850978166528 (Original Log Time 2024/11/19-15:19:32.947445) EVENT_LOG_v1 {"time_micros": 1732000772947429, "job": 121, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4643, 4645], "score": 5.69623, "input_data_size": 25037137, "oldest_snapshot_seqno": 123901122} 2024/11/19-15:19:33.222054 139850978166528 (Original Log Time 2024/11/19-15:19:33.221662) [compaction_job.cc:2022] [default] [JOB 121] Compacted 2@1 files to L2 => 12828931 bytes 2024/11/19-15:19:33.222057 139850978166528 (Original Log Time 2024/11/19-15:19:33.221942) [compaction_job.cc:1224] [default] compacted to: files[1 105 32 14 0 0 0] max score 5.40 (117 files need compaction), MB/sec: 91.5 rd, 46.9 wr, level 2, files in(2, 0) out(2 +0 blob) MB in(23.9, 0.0 +0.0 blob) out(12.2 +0.0 blob), read-write-amplify(1.5) write-amplify(0.5) OK, records in: 624000, records dropped: 0 output_compression: Snappy 2024/11/19-15:19:33.222071 139850978166528 (Original Log Time 2024/11/19-15:19:33.221962) EVENT_LOG_v1 {"time_micros": 1732000773221952, "cf": "default", "job": 121, "event": "compaction_finished", "compaction_time_micros": 273600, "compaction_time_cpu_micros": 266074, "output_level": 2, "num_output_files": 2, "total_output_size": 12828931, "num_input_records": 624000, "num_output_records": 624000, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 105, 32, 14, 0, 0, 0]} 2024/11/19-15:19:33.222138 139850978166528 [compaction_job.cc:2437] [default] [JOB 123] Compacting 4@1 files to L2, score 5.40, subcompactions 1 : 1 2024/11/19-15:19:33.222146 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 121 Base level 1, inputs: [4661(31MB) 4663(31MB) 4665(31MB) 4667(31MB)] 2024/11/19-15:19:33.405336 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-01 2024/11/19-15:19:33.405408 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00116/att-01/compact.done) = No such file or directory: 11.212 sec on http probe failed: 5, giveup 2024/11/19-15:19:33.405425 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00116/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.238 sec, real 11.529 (submit 311.970 ms wait 11.212 s), input 127.238 MiB, send shutdown 2024/11/19-15:19:33.411190 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-01 2024/11/19-15:19:33.411391 139851137550080 [dcompact_etcd.cc:1773] job-00116/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:33.417283 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-01 2024/11/19-15:19:33.417351 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00119/att-01/compact.done) = No such file or directory: 4.585 sec on http probe failed: 4, retry 2024/11/19-15:19:33.505572 139850978166528 [dcompact_etcd.cc:1163] job-00123/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.283 sec, input = 127.819 MiB 2024/11/19-15:19:33.710249 139851137550080 [dcompact_etcd.cc:1163] job-00116/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.299 sec, input = 127.238 MiB 2024/11/19-15:19:33.790832 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-02 2024/11/19-15:19:33.790911 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00120/att-02/compact.done) = No such file or directory: 2.905 sec on http probe failed: 4, retry 2024/11/19-15:19:34.275773 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-00 2024/11/19-15:19:34.275862 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00122/att-00/compact.done) = No such file or directory: 5.208 sec on http probe failed: 2, retry 2024/11/19-15:19:34.474628 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-02 2024/11/19-15:19:34.474721 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00120/att-02/compact.done) = No such file or directory: 3.590 sec on http probe failed: 5, giveup 2024/11/19-15:19:34.474738 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00120/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 33.874 sec, real 3.877 (submit 280.402 ms wait 3.590 s), input 33.874 MiB, send shutdown 2024/11/19-15:19:34.483154 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00120/att-02 2024/11/19-15:19:34.483391 139851148035840 [dcompact_etcd.cc:1773] job-00120/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:34.485179 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00120) = success 2024/11/19-15:19:34.485190 139851148035840 [dcompact_etcd.cc:1773] job-00120/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:19:34.485209 139851148035840 (Original Log Time 2024/11/19-15:19:22.810615) EVENT_LOG_v1 {"time_micros": 1732000762810607, "job": 120, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4653, 4632, 4641], "score": 5.7667, "input_data_size": 37836669, "oldest_snapshot_seqno": 123795481} 2024/11/19-15:19:34.485217 139851148035840 [compaction_job.cc:2437] [default] [JOB 120] Compacting 3@1 files to L2, score 5.77, subcompactions 1 : 1 2024/11/19-15:19:34.485225 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 118 Base level 1, inputs: [4653(2514KB) 4632(31MB) 4641(2077KB)] 2024/11/19-15:19:34.485498 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004757.sst) /storage/mytopling/..//tmp/Topling-82038-6SnpAA 2024/11/19-15:19:34.500751 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-01 2024/11/19-15:19:34.500834 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00119/att-01/compact.done) = No such file or directory: 5.666 sec on http probe failed: 5, giveup 2024/11/19-15:19:34.500855 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00119/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 53.697 sec, real 5.941 (submit 265.484 ms wait 5.666 s), input 53.697 MiB, send shutdown 2024/11/19-15:19:34.508150 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-01 2024/11/19-15:19:34.508376 139851158521600 [dcompact_etcd.cc:1773] job-00119/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:34.618478 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004757.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.036 GB 2024/11/19-15:19:34.618606 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004757.sst): index pass time = 0.00's,37845.190'MB/sec index type = UintIndex_AllOne usrkeys = 480000 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0038 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:19:34.657973 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004757.sst): first pass time = 0.17's, 205.956'MB/sec 2024/11/19-15:19:34.658019 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004757.sst): old prealloc_size = 41620335, real_size = 27540360 2024/11/19-15:19:34.658041 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004757.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 480000 prefix = 4 raw-key = 0.0038 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0259 GB zip-val = 0.0274 GB avg-val = 54.00 avg-zval = 57.12 2024/11/19-15:19:34.703765 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004757.sst): second pass time = 0.00's,57727772.829'MB/sec, value only(73.0% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 599.648'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.191'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 7.392'MB/sec write SST data time = 0.03's, 930.244'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.026'GB zip my value throughput = 57727772.829'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 480000 avg-key = 20.00 avg-zkey = 0.00 avg-val = 54.00 avg-zval = 57.12 usrkeys = 480000 avg-key = 12.00 avg-zkey = 0.00 avg-val = 54.00 avg-zval = 57.12 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0058 GB value = 0.0259 GB dict = 0.00 MB all = 0.0355 GB } __ZipSize{ index = 0.0000 GB value = 0.0274 GB dict = 0.00 MB all = 0.0274 GB } UnZip/Zip{ index =30000.0000 value = 0.9453 dict = 0.00 all = 1.2953 } Zip/UnZip{ index = 0.0000 value = 1.0579 dict = 0.00 all = 0.7720 } ---------------------------- total value len = 5.176918 GB avg = 0.289 KB (by entry num) total key len = 0.449963 GB avg = 0.025 KB total ukey len = 0.306544 GB avg = 0.017 KB total ukey num = 0.017927332 Billion total entry num = 0.017927332 Billion write speed all = 97.459487898 MB/sec (with seq num) write speed all = 94.975428308 MB/sec (without seq num) 2024/11/19-15:19:34.705367 139851148035840 [compaction_job.cc:1948] [default] [JOB 120] Generated table #4757: 480000 keys, 27421320 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:34.705419 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000774705395, "cf_name": "default", "job": 120, "event": "table_file_creation", "file_number": 4757, "file_size": 27421320, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27419984, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 9600000, "raw_average_key_size": 20, "raw_value_size": 25919770, "raw_average_value_size": 53, "num_data_blocks": 1, "num_entries": 480000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000774, "oldest_key_time": 0, "file_creation_time": 1732000774, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4757, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:34.712432 139851148035840 [compaction_job.cc:703] job-00120: subcompact[0], size: 0.027421 G, files: /storage/mytopling/.rocksdb [004757] 2024/11/19-15:19:34.713136 139851148035840 (Original Log Time 2024/11/19-15:19:34.485257) EVENT_LOG_v1 {"time_micros": 1732000774485238, "job": 120, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4653, 4632, 4641], "score": 5.7667, "input_data_size": 37836669, "oldest_snapshot_seqno": 123795481} 2024/11/19-15:19:34.713145 139851148035840 (Original Log Time 2024/11/19-15:19:34.712742) [compaction_job.cc:2022] [default] [JOB 120] Compacted 3@1 files to L2 => 27421320 bytes 2024/11/19-15:19:34.713149 139851148035840 (Original Log Time 2024/11/19-15:19:34.713030) [compaction_job.cc:1224] [default] compacted to: files[1 102 33 14 0 0 0] max score 5.15 (114 files need compaction), MB/sec: 166.5 rd, 120.7 wr, level 2, files in(3, 0) out(1 +0 blob) MB in(36.1, 0.0 +0.0 blob) out(26.2 +0.0 blob), read-write-amplify(1.7) write-amplify(0.7) OK, records in: 480000, records dropped: 0 output_compression: Snappy 2024/11/19-15:19:34.713162 139851148035840 (Original Log Time 2024/11/19-15:19:34.713054) EVENT_LOG_v1 {"time_micros": 1732000774713043, "cf": "default", "job": 120, "event": "compaction_finished", "compaction_time_micros": 227187, "compaction_time_cpu_micros": 226187, "output_level": 2, "num_output_files": 1, "total_output_size": 27421320, "num_input_records": 480000, "num_output_records": 480000, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 102, 33, 14, 0, 0, 0]} 2024/11/19-15:19:34.713228 139851148035840 [compaction_job.cc:2437] [default] [JOB 124] Compacting 4@1 files to L2, score 5.15, subcompactions 1 : 1 2024/11/19-15:19:34.713237 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 122 Base level 1, inputs: [4670(31MB) 4673(9648KB) 4633(31MB) 4639(31MB)] 2024/11/19-15:19:34.828803 139851158521600 [dcompact_etcd.cc:1163] job-00119/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.320 sec, input = 53.697 MiB 2024/11/19-15:19:34.936845 139852349699840 [db_impl_write.cc:2181] [default] New memtable created with log file: #4758. Immutable memtables: 0. 2024/11/19-15:19:34.937463 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 125] Syncing log #4745 2024/11/19-15:19:34.937564 139850967680768 (Original Log Time 2024/11/19-15:19:34.937257) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:19:34.937574 139850967680768 [flush_job.cc:879] [default] [JOB 125] Flushing memtable with next log file: 4758 2024/11/19-15:19:34.937599 139850967680768 EVENT_LOG_v1 {"time_micros": 1732000774937592, "job": 125, "event": "flush_started", "num_memtables": 1, "num_entries": 1437596, "num_deletes": 0, "total_data_size": 488645785, "memory_usage": 541851016, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:19:34.937605 139850967680768 [flush_job.cc:910] [default] [JOB 125] Level-0 flush table #4759: started 2024/11/19-15:19:34.938273 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000013.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004759.sst in ConvertToSST, mem_size = 516.749 M, time = 656.595 us 2024/11/19-15:19:35.002000 139851148035840 [dcompact_etcd.cc:1163] job-00124/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.288 sec, input = 105.246 MiB 2024/11/19-15:19:35.104040 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-02 2024/11/19-15:19:35.104181 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00111/att-02/compact.done) = No such file or directory: 9.016 sec on http probe failed: 3, retry 2024/11/19-15:19:35.168387 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-01 2024/11/19-15:19:35.168510 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00117/att-01/compact.done) = No such file or directory: 11.215 sec on http probe failed: 5, giveup 2024/11/19-15:19:35.168540 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00117/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.241 sec, real 11.653 (submit 428.848 ms wait 11.215 s), input 127.241 MiB, send shutdown 2024/11/19-15:19:35.173251 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-01 2024/11/19-15:19:35.186237 139851169007360 [dcompact_etcd.cc:1773] job-00117/att-01: CleanFiles 0.013 sec : 0.000 2024/11/19-15:19:35.393988 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004759.sst): fsize = 516.751 M, time(ms): open: 0.043, seek: 0.020, finish: 0.048, meta: 0.013, Flush: 0.018, sync: 455.538, close: 0.012, all: 455.692 2024/11/19-15:19:35.395066 139850967680768 [flush_job.cc:1083] [default] [JOB 125] Flush lasted 457502 microseconds, and 366044 cpu microseconds. 2024/11/19-15:19:35.416409 139850967680768 (Original Log Time 2024/11/19-15:19:35.394036) [flush_job.cc:1029] [default] [JOB 125] Level-0 flush table #4759: 541852442 bytes OK (needs compaction) 2024/11/19-15:19:35.416420 139850967680768 (Original Log Time 2024/11/19-15:19:35.395083) [memtable_list.cc:560] [default] Level-0 commit table #4759 started 2024/11/19-15:19:35.416424 139850967680768 (Original Log Time 2024/11/19-15:19:35.416336) [memtable_list.cc:759] [default] Level-0 commit table #4759: memtable #1 done 2024/11/19-15:19:35.416427 139850967680768 (Original Log Time 2024/11/19-15:19:35.416361) EVENT_LOG_v1 {"time_micros": 1732000775416355, "job": 125, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [2, 102, 33, 14, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:19:35.416430 139850967680768 (Original Log Time 2024/11/19-15:19:35.416381) [db_impl_compaction_flush.cc:310] [default] Level summary: files[2 102 33 14 0 0 0] max score 4.94 (111 files need compaction) 2024/11/19-15:19:35.502093 139851169007360 [dcompact_etcd.cc:1163] job-00117/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.316 sec, input = 127.241 MiB 2024/11/19-15:19:36.180454 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-02 2024/11/19-15:19:36.180631 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00119/att-02/compact.done) = No such file or directory: 1.343 sec on http probe failed: 1, retry 2024/11/19-15:19:36.284009 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-00 2024/11/19-15:19:36.284107 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00122/att-00/compact.done) = No such file or directory: 7.214 sec on http probe failed: 3, retry 2024/11/19-15:19:36.706346 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-00 2024/11/19-15:19:36.706435 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00123/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:19:36.899403 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-02 2024/11/19-15:19:36.899504 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00116/att-02/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:19:37.113134 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-02 2024/11/19-15:19:37.113229 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00111/att-02/compact.done) = No such file or directory: 11.022 sec on http probe failed: 4, retry 2024/11/19-15:19:37.264082 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-02 2024/11/19-15:19:37.264154 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00119/att-02/compact.done) = No such file or directory: 2.426 sec on http probe failed: 2, retry 2024/11/19-15:19:37.638695 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-00 2024/11/19-15:19:37.638806 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00124/att-00/compact.done) = No such file or directory: 2.631 sec on http probe failed: 1, retry 2024/11/19-15:19:38.292687 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-00 2024/11/19-15:19:38.292786 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00122/att-00/compact.done) = No such file or directory: 9.222 sec on http probe failed: 4, retry 2024/11/19-15:19:38.346223 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-02 2024/11/19-15:19:38.346312 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00119/att-02/compact.done) = No such file or directory: 3.509 sec on http probe failed: 3, retry 2024/11/19-15:19:38.692278 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-02 2024/11/19-15:19:38.692365 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00117/att-02/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:19:38.715132 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-00 2024/11/19-15:19:38.715201 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00123/att-00/compact.done) = No such file or directory: 5.201 sec on http probe failed: 2, retry 2024/11/19-15:19:38.906168 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-02 2024/11/19-15:19:38.906241 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00116/att-02/compact.done) = No such file or directory: 5.189 sec on http probe failed: 2, retry 2024/11/19-15:19:39.121718 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-02 2024/11/19-15:19:39.121846 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00111/att-02/compact.done) = No such file or directory: 13.031 sec on http probe failed: 5, giveup 2024/11/19-15:19:39.121871 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00111/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 241.761 sec, real 13.333 (submit 293.321 ms wait 13.031 s), input 241.761 MiB, send shutdown 2024/11/19-15:19:39.130381 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00111/att-02 2024/11/19-15:19:39.130659 139851179493120 [dcompact_etcd.cc:1773] job-00111/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:39.130941 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00111) = success 2024/11/19-15:19:39.130952 139851179493120 [dcompact_etcd.cc:1773] job-00111/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:39.130978 139851179493120 (Original Log Time 2024/11/19-15:18:59.075873) EVENT_LOG_v1 {"time_micros": 1732000739075863, "job": 111, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4590, 4593, 4594, 4597], "score": 1.01385, "input_data_size": 209668640, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:39.130989 139851179493120 [compaction_job.cc:2437] [default] [JOB 111] Compacting 4@2 files to L3, score 1.01, subcompactions 1 : 1 2024/11/19-15:19:39.131002 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 109 Base level 2, inputs: [4590(41MB) 4593(52MB) 4594(52MB) 4597(52MB)] 2024/11/19-15:19:39.131413 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004760.sst) /storage/mytopling/..//tmp/Topling-82038-qIbZ8n 2024/11/19-15:19:39.428748 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-02 2024/11/19-15:19:39.428847 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00119/att-02/compact.done) = No such file or directory: 4.592 sec on http probe failed: 4, retry 2024/11/19-15:19:39.495420 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004760.sst): first pass time = 0.36's, 368.742'MB/sec 2024/11/19-15:19:39.495422 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004760.sst: sumWaitingMem = 0.012 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0121 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:19:39.495455 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004760.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.012 GB, dictZip workingMem = 0.0065 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:19:39.579039 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004760.sst): index pass time = 0.08's, 177.454'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 926961 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:19:39.648468 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-00 2024/11/19-15:19:39.648589 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00124/att-00/compact.done) = No such file or directory: 4.637 sec on http probe failed: 2, retry 2024/11/19-15:19:40.308000 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-00 2024/11/19-15:19:40.308113 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00122/att-00/compact.done) = No such file or directory: 11.231 sec on http probe failed: 5, giveup 2024/11/19-15:19:40.308141 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00122/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.823 sec, real 11.526 (submit 279.862 ms wait 11.231 s), input 127.823 MiB, send shutdown 2024/11/19-15:19:40.320273 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-00 2024/11/19-15:19:40.320555 139851127064320 [dcompact_etcd.cc:1773] job-00122/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:40.518488 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-02 2024/11/19-15:19:40.518582 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00119/att-02/compact.done) = No such file or directory: 5.681 sec on http probe failed: 5, giveup 2024/11/19-15:19:40.518605 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00119/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 53.697 sec, real 6.010 (submit 320.309 ms wait 5.681 s), input 53.697 MiB, send shutdown 2024/11/19-15:19:40.528762 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00119/att-02 2024/11/19-15:19:40.529953 139851158521600 [dcompact_etcd.cc:1773] job-00119/att-02: CleanFiles 0.001 sec : 0.000 2024/11/19-15:19:40.530163 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00119) = success 2024/11/19-15:19:40.530175 139851158521600 [dcompact_etcd.cc:1773] job-00119/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:40.530200 139851158521600 (Original Log Time 2024/11/19-15:19:22.556740) EVENT_LOG_v1 {"time_micros": 1732000762556734, "job": 119, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4650, 4651], "score": 5.87214, "input_data_size": 56605120, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:40.530210 139851158521600 [compaction_job.cc:2437] [default] [JOB 119] Compacting 2@1 files to L2, score 5.87, subcompactions 1 : 1 2024/11/19-15:19:40.530220 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 117 Base level 1, inputs: [4650(31MB) 4651(22MB)] 2024/11/19-15:19:40.530653 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004761.sst) /storage/mytopling/..//tmp/Topling-82038-5YrtOf 2024/11/19-15:19:40.585441 139851127064320 [dcompact_etcd.cc:1163] job-00122/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.265 sec, input = 127.823 MiB 2024/11/19-15:19:40.694951 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004761.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.006 GB, nltTrie workingMem = 0.0008 GB, waited 0.000 sec, Key+Value = 0.056 GB 2024/11/19-15:19:40.702553 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004761.sst): index pass time = 0.01's, 133.565'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 84154 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0001 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:19:40.706151 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-02 2024/11/19-15:19:40.706240 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00117/att-02/compact.done) = No such file or directory: 5.193 sec on http probe failed: 2, retry 2024/11/19-15:19:40.728491 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-00 2024/11/19-15:19:40.728571 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00123/att-00/compact.done) = No such file or directory: 7.215 sec on http probe failed: 3, retry 2024/11/19-15:19:40.771194 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004761.sst): first pass time = 0.24's, 234.093'MB/sec 2024/11/19-15:19:40.771247 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004761.sst): old prealloc_size = 62265632, real_size = 54723088 2024/11/19-15:19:40.771309 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004761.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 84154 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0001 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0543 GB zip-val = 0.0546 GB avg-val = 645.07 avg-zval = 648.32 2024/11/19-15:19:40.771390 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004761.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0002 GB, waited 0.076 sec, Key+Value = 0.056 GB 2024/11/19-15:19:40.830759 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004761.sst): second pass time = 0.00's,106650660.118'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 919.431'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2411.347'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.529'MB/sec write SST data time = 0.02's, 2784.454'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 106650660.118'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 84154 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.07 avg-zval = 648.32 usrkeys = 84154 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.07 avg-zval = 648.32 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6664 UnZipSize{ index = 0.0013 GB value = 0.0543 GB dict = 0.00 MB all = 0.0563 GB } __ZipSize{ index = 0.0001 GB value = 0.0546 GB dict = 0.00 MB all = 0.0547 GB } UnZip/Zip{ index = 9.4174 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1062 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 5.231204 GB avg = 0.290 KB (by entry num) total key len = 0.451983 GB avg = 0.025 KB total ukey len = 0.307891 GB avg = 0.017 KB total ukey num = 0.018011486 Billion total entry num = 0.018011486 Billion write speed all = 88.990855619 MB/sec (with seq num) write speed all = 86.734575478 MB/sec (without seq num) 2024/11/19-15:19:40.834055 139851158521600 [compaction_job.cc:1948] [default] [JOB 119] Generated table #4761: 84154 keys, 54703820 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:40.834107 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000780834083, "cf_name": "default", "job": 119, "event": "table_file_creation", "file_number": 4761, "file_size": 54703820, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54558912, "index_size": 142976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2019696, "raw_average_key_size": 24, "raw_value_size": 54285186, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 84154, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000780, "oldest_key_time": 0, "file_creation_time": 1732000780, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4761, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:40.847819 139851158521600 [compaction_job.cc:703] job-00119: subcompact[0], size: 0.054704 G, files: /storage/mytopling/.rocksdb [004761] 2024/11/19-15:19:40.848503 139851158521600 (Original Log Time 2024/11/19-15:19:40.530340) EVENT_LOG_v1 {"time_micros": 1732000780530236, "job": 119, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4650, 4651], "score": 5.87214, "input_data_size": 56605120, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:40.848513 139851158521600 (Original Log Time 2024/11/19-15:19:40.848129) [compaction_job.cc:2022] [default] [JOB 119] Compacted 2@1 files to L2 => 54703820 bytes 2024/11/19-15:19:40.848516 139851158521600 (Original Log Time 2024/11/19-15:19:40.848381) [compaction_job.cc:1224] [default] compacted to: files[2 100 34 14 0 0 0] max score 4.94 (112 files need compaction), MB/sec: 178.3 rd, 172.3 wr, level 2, files in(2, 0) out(1 +0 blob) MB in(54.0, 0.0 +0.0 blob) out(52.2 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 84154, records dropped: 0 output_compression: Snappy 2024/11/19-15:19:40.848534 139851158521600 (Original Log Time 2024/11/19-15:19:40.848403) EVENT_LOG_v1 {"time_micros": 1732000780848392, "cf": "default", "job": 119, "event": "compaction_finished", "compaction_time_micros": 317489, "compaction_time_cpu_micros": 221725, "output_level": 2, "num_output_files": 1, "total_output_size": 54703820, "num_input_records": 84154, "num_output_records": 84154, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 100, 34, 14, 0, 0, 0]} 2024/11/19-15:19:40.848600 139851158521600 [compaction_job.cc:2437] [default] [JOB 126] Compacting 4@1 files to L2, score 4.94, subcompactions 1 : 1 2024/11/19-15:19:40.848608 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 124 Base level 1, inputs: [4646(31MB) 4649(31MB) 4654(31MB) 4656(31MB)] 2024/11/19-15:19:40.911233 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-02 2024/11/19-15:19:40.911301 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00116/att-02/compact.done) = No such file or directory: 7.196 sec on http probe failed: 3, retry 2024/11/19-15:19:41.068332 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004760.sst): old prealloc_size = 147639500, real_size = 36392208 2024/11/19-15:19:41.068801 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004760.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926961 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.79 avg-zval = 37.11 2024/11/19-15:19:41.068895 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004760.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 1.573 sec, Key+Value = 0.134 GB 2024/11/19-15:19:41.176321 139851158521600 [dcompact_etcd.cc:1163] job-00126/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.327 sec, input = 127.819 MiB 2024/11/19-15:19:41.608472 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004760.sst): second pass time = 1.57's, 68.833'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.54's, 68.965'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3925.135'MB/sec (index lex order gen) rebuild zvType time = 0.44's, 0.528'MB/sec write SST data time = 0.10's, 370.990'MB/sec dict compress time = 0.00's, 1394.939'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 10777.054'MB/sec zip my value time = 1.57's, unzip length = 0.108'GB zip my value throughput = 68.833'MB/sec zip pipeline throughput = 68.833'MB/sec entries = 926961 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 37.11 usrkeys = 926961 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 37.11 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 10 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5126 value = 3.1475 dict = 1.00 all = 3.6031 } Zip/UnZip{ index = 0.0951 value = 0.3177 dict = 1.00 all = 0.2775 } ---------------------------- total value len = 5.339466 GB avg = 0.282 KB (by entry num) total key len = 0.477938 GB avg = 0.025 KB total ukey len = 0.326430 GB avg = 0.017 KB total ukey num = 0.018938447 Billion total entry num = 0.018938447 Billion write speed all = 89.996552945 MB/sec (with seq num) write speed all = 87.652696593 MB/sec (without seq num) 2024/11/19-15:19:41.609756 139851179493120 [compaction_job.cc:1948] [default] [JOB 111] Generated table #4760: 926961 keys, 37250159 bytes, temperature: kUnknown 2024/11/19-15:19:41.609806 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000781609783, "cf_name": "default", "job": 111, "event": "table_file_creation", "file_number": 4760, "file_size": 37250159, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34396736, "index_size": 1763520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954908, "raw_average_key_size": 28, "raw_value_size": 108262898, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926961, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000779, "oldest_key_time": 0, "file_creation_time": 1732000779, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4760, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:41.632202 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004762.sst) /storage/mytopling/..//tmp/Topling-82038-lcb72p 2024/11/19-15:19:41.656732 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-00 2024/11/19-15:19:41.656804 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00124/att-00/compact.done) = No such file or directory: 6.647 sec on http probe failed: 3, retry 2024/11/19-15:19:41.929078 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004762.sst): first pass time = 0.30's, 401.834'MB/sec 2024/11/19-15:19:41.929081 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004762.sst: sumWaitingMem = 0.011 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0107 GB, waited 0.000 sec, Key+Value = 0.119 GB 2024/11/19-15:19:41.929115 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004762.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.011 GB, dictZip workingMem = 0.0058 GB, waited 0.000 sec, Key+Value = 0.119 GB 2024/11/19-15:19:41.994293 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004762.sst): index pass time = 0.07's, 202.223'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 823699 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0132 GB zip-key = 0.0016 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:19:42.718546 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-02 2024/11/19-15:19:42.718642 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00117/att-02/compact.done) = No such file or directory: 7.204 sec on http probe failed: 3, retry 2024/11/19-15:19:42.740147 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-00 2024/11/19-15:19:42.740231 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00123/att-00/compact.done) = No such file or directory: 9.223 sec on http probe failed: 4, retry 2024/11/19-15:19:42.927205 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-02 2024/11/19-15:19:42.927285 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00116/att-02/compact.done) = No such file or directory: 9.204 sec on http probe failed: 4, retry 2024/11/19-15:19:43.243429 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004762.sst): old prealloc_size = 147639500, real_size = 32300608 2024/11/19-15:19:43.243870 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004762.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 823699 prefix = 4 raw-key = 0.0132 GB zip-key = 0.0016 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0962 GB zip-val = 0.0305 GB avg-val = 116.82 avg-zval = 37.07 2024/11/19-15:19:43.244009 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004762.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0021 GB, waited 1.315 sec, Key+Value = 0.119 GB 2024/11/19-15:19:43.662942 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-00 2024/11/19-15:19:43.663035 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00124/att-00/compact.done) = No such file or directory: 8.655 sec on http probe failed: 4, retry 2024/11/19-15:19:43.718266 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004762.sst): second pass time = 1.31's, 73.214'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.47's, 69.633'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3635.397'MB/sec (index lex order gen) rebuild zvType time = 0.39's, 0.535'MB/sec write SST data time = 0.09's, 370.159'MB/sec dict compress time = 0.00's, 1340.929'MB/sec z-dict build time = 0.00's, sample length = 0.960'MB, throughput = 10648.890'MB/sec zip my value time = 1.31's, unzip length = 0.096'GB zip my value throughput = 73.214'MB/sec zip pipeline throughput = 73.214'MB/sec entries = 823699 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.82 avg-zval = 37.07 usrkeys = 823699 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.82 avg-zval = 37.07 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0165 GB value = 0.0962 GB dict = 0.96 MB all = 0.1193 GB } __ZipSize{ index = 0.0016 GB value = 0.0305 GB dict = 0.96 MB all = 0.0331 GB } UnZip/Zip{ index = 10.5811 value = 3.1510 dict = 1.00 all = 3.6078 } Zip/UnZip{ index = 0.0945 value = 0.3174 dict = 1.00 all = 0.2772 } ---------------------------- total value len = 5.435690 GB avg = 0.275 KB (by entry num) total key len = 0.501001 GB avg = 0.025 KB total ukey len = 0.342904 GB avg = 0.017 KB total ukey num = 0.019762146 Billion total entry num = 0.019762146 Billion write speed all = 88.939064517 MB/sec (with seq num) write speed all = 86.570571095 MB/sec (without seq num) 2024/11/19-15:19:43.719241 139851179493120 [compaction_job.cc:1948] [default] [JOB 111] Generated table #4762: 823699 keys, 33063731 bytes, temperature: kUnknown 2024/11/19-15:19:43.719294 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783719269, "cf_name": "default", "job": 111, "event": "table_file_creation", "file_number": 4762, "file_size": 33063731, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 30537584, "index_size": 1556928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 23063572, "raw_average_key_size": 28, "raw_value_size": 96223586, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 823699, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000781, "oldest_key_time": 0, "file_creation_time": 1732000781, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4762, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:43.738908 139851179493120 [compaction_job.cc:703] job-00111: subcompact[0], size: 0.070314 G, files: /storage/mytopling/.rocksdb [004760,004762] 2024/11/19-15:19:43.741348 139851179493120 (Original Log Time 2024/11/19-15:19:39.131041) EVENT_LOG_v1 {"time_micros": 1732000779131017, "job": 111, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4590, 4593, 4594, 4597], "score": 1.01385, "input_data_size": 209668640, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:43.741358 139851179493120 (Original Log Time 2024/11/19-15:19:43.740952) [compaction_job.cc:2022] [default] [JOB 111] Compacted 4@2 files to L3 => 70313890 bytes 2024/11/19-15:19:43.741362 139851179493120 (Original Log Time 2024/11/19-15:19:43.741203) [compaction_job.cc:1224] [default] compacted to: files[2 100 30 16 0 0 0] max score 4.70 (108 files need compaction), MB/sec: 45.5 rd, 15.3 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(200.0, 0.0 +0.0 blob) out(67.1 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 1750660, records dropped: 0 output_compression: Snappy 2024/11/19-15:19:43.741367 139851179493120 (Original Log Time 2024/11/19-15:19:43.741227) EVENT_LOG_v1 {"time_micros": 1732000783741216, "cf": "default", "job": 111, "event": "compaction_finished", "compaction_time_micros": 4607887, "compaction_time_cpu_micros": 3662261, "output_level": 3, "num_output_files": 2, "total_output_size": 70313890, "num_input_records": 1750660, "num_output_records": 1750660, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 100, 30, 16, 0, 0, 0]} 2024/11/19-15:19:43.745980 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004548.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.746006 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783746003, "job": 111, "event": "table_file_deletion", "file_number": 4548} 2024/11/19-15:19:43.750313 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004549.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.750333 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783750331, "job": 111, "event": "table_file_deletion", "file_number": 4549} 2024/11/19-15:19:43.754396 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004550.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.754416 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783754414, "job": 111, "event": "table_file_deletion", "file_number": 4550} 2024/11/19-15:19:43.758767 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004551.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.758786 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783758784, "job": 111, "event": "table_file_deletion", "file_number": 4551} 2024/11/19-15:19:43.763192 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004553.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.763211 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783763209, "job": 111, "event": "table_file_deletion", "file_number": 4553} 2024/11/19-15:19:43.767354 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004555.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.767373 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783767371, "job": 111, "event": "table_file_deletion", "file_number": 4555} 2024/11/19-15:19:43.771795 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004556.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.771814 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783771812, "job": 111, "event": "table_file_deletion", "file_number": 4556} 2024/11/19-15:19:43.775906 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004557.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.775931 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783775929, "job": 111, "event": "table_file_deletion", "file_number": 4557} 2024/11/19-15:19:43.780087 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004558.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.780106 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783780104, "job": 111, "event": "table_file_deletion", "file_number": 4558} 2024/11/19-15:19:43.784716 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004559.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.784737 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783784735, "job": 111, "event": "table_file_deletion", "file_number": 4559} 2024/11/19-15:19:43.789071 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004560.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.789091 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783789089, "job": 111, "event": "table_file_deletion", "file_number": 4560} 2024/11/19-15:19:43.793411 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004561.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.793430 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783793428, "job": 111, "event": "table_file_deletion", "file_number": 4561} 2024/11/19-15:19:43.794046 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-01 2024/11/19-15:19:43.794138 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00122/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:19:43.797709 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004562.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.797729 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783797727, "job": 111, "event": "table_file_deletion", "file_number": 4562} 2024/11/19-15:19:43.801864 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004563.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.801883 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783801881, "job": 111, "event": "table_file_deletion", "file_number": 4563} 2024/11/19-15:19:43.806157 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004565.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.806177 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783806175, "job": 111, "event": "table_file_deletion", "file_number": 4565} 2024/11/19-15:19:43.810493 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004566.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.810512 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783810510, "job": 111, "event": "table_file_deletion", "file_number": 4566} 2024/11/19-15:19:43.814585 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004569.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.814603 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783814601, "job": 111, "event": "table_file_deletion", "file_number": 4569} 2024/11/19-15:19:43.818834 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004570.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.818852 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783818850, "job": 111, "event": "table_file_deletion", "file_number": 4570} 2024/11/19-15:19:43.823005 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004571.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.823023 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783823022, "job": 111, "event": "table_file_deletion", "file_number": 4571} 2024/11/19-15:19:43.824482 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004572.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.824499 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783824497, "job": 111, "event": "table_file_deletion", "file_number": 4572} 2024/11/19-15:19:43.892299 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004574.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.892345 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783892341, "job": 111, "event": "table_file_deletion", "file_number": 4574} 2024/11/19-15:19:43.970020 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004579.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:43.970063 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000783970059, "job": 111, "event": "table_file_deletion", "file_number": 4579} 2024/11/19-15:19:44.040565 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004586.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:44.040607 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000784040603, "job": 111, "event": "table_file_deletion", "file_number": 4586} 2024/11/19-15:19:44.047165 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004588.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:44.047192 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000784047190, "job": 111, "event": "table_file_deletion", "file_number": 4588} 2024/11/19-15:19:44.053819 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004589.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:44.053844 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000784053841, "job": 111, "event": "table_file_deletion", "file_number": 4589} 2024/11/19-15:19:44.123738 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004596.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:44.123782 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000784123778, "job": 111, "event": "table_file_deletion", "file_number": 4596} 2024/11/19-15:19:44.192724 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004605.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:44.192771 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000784192767, "job": 111, "event": "table_file_deletion", "file_number": 4605} 2024/11/19-15:19:44.266327 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004627.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:44.266379 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000784266375, "job": 111, "event": "table_file_deletion", "file_number": 4627} 2024/11/19-15:19:44.266561 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004634.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:44.266575 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000784266573, "job": 111, "event": "table_file_deletion", "file_number": 4634} 2024/11/19-15:19:44.266732 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004635.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:44.266743 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000784266742, "job": 111, "event": "table_file_deletion", "file_number": 4635} 2024/11/19-15:19:44.266942 139851179493120 [compaction_job.cc:2437] [default] [JOB 127] Compacting 4@1 files to L2, score 4.70, subcompactions 1 : 1 2024/11/19-15:19:44.266951 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 125 Base level 1, inputs: [4658(31MB) 4660(31MB) 4662(31MB) 4664(31MB)] 2024/11/19-15:19:44.380200 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-00 2024/11/19-15:19:44.380289 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00126/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:19:44.578548 139851179493120 [dcompact_etcd.cc:1163] job-00127/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.311 sec, input = 127.816 MiB 2024/11/19-15:19:44.725067 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-02 2024/11/19-15:19:44.725175 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00117/att-02/compact.done) = No such file or directory: 9.217 sec on http probe failed: 4, retry 2024/11/19-15:19:44.745890 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-00 2024/11/19-15:19:44.745990 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00123/att-00/compact.done) = No such file or directory: 11.235 sec on http probe failed: 5, giveup 2024/11/19-15:19:44.746009 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00123/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.819 sec, real 11.523 (submit 282.886 ms wait 11.235 s), input 127.819 MiB, send shutdown 2024/11/19-15:19:44.754049 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-00 2024/11/19-15:19:44.754285 139850978166528 [dcompact_etcd.cc:1773] job-00123/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:44.935799 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-02 2024/11/19-15:19:44.935899 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00116/att-02/compact.done) = No such file or directory: 11.217 sec on http probe failed: 5, giveup 2024/11/19-15:19:44.935918 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00116/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.238 sec, real 11.524 (submit 298.786 ms wait 11.217 s), input 127.238 MiB, send shutdown 2024/11/19-15:19:44.944663 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00116/att-02 2024/11/19-15:19:44.944887 139851137550080 [dcompact_etcd.cc:1773] job-00116/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:44.945130 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00116) = success 2024/11/19-15:19:44.945136 139851137550080 [dcompact_etcd.cc:1773] job-00116/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:44.945155 139851137550080 (Original Log Time 2024/11/19-15:19:10.354630) EVENT_LOG_v1 {"time_micros": 1732000750354625, "job": 116, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4636, 4637, 4638, 4640], "score": 6.37143, "input_data_size": 134026816, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:44.945162 139851137550080 [compaction_job.cc:2437] [default] [JOB 116] Compacting 4@1 files to L2, score 6.37, subcompactions 1 : 1 2024/11/19-15:19:44.945169 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 114 Base level 1, inputs: [4636(31MB) 4637(31MB) 4638(31MB) 4640(31MB)] 2024/11/19-15:19:44.945433 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004763.sst) /storage/mytopling/..//tmp/Topling-82038-uJxBNN 2024/11/19-15:19:45.007719 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004763.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:19:45.016141 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004763.sst): index pass time = 0.01's, 143.586'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 100309 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:19:45.053720 139850978166528 [dcompact_etcd.cc:1163] job-00123/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.299 sec, input = 127.819 MiB 2024/11/19-15:19:45.083287 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004763.sst): first pass time = 0.14's, 486.858'MB/sec 2024/11/19-15:19:45.083336 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004763.sst): old prealloc_size = 73819750, real_size = 65223600 2024/11/19-15:19:45.083404 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004763.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 100309 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0650 GB avg-val = 645.03 avg-zval = 648.28 2024/11/19-15:19:45.083478 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004763.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.076 sec, Key+Value = 0.067 GB 2024/11/19-15:19:45.154696 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004763.sst): second pass time = 0.00's,249814810.811'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 913.883'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2682.816'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.544'MB/sec write SST data time = 0.03's, 2590.289'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 249814810.811'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 100309 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.03 avg-zval = 648.28 usrkeys = 100309 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.03 avg-zval = 648.28 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7943 UnZipSize{ index = 0.0016 GB value = 0.0647 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0002 GB value = 0.0650 GB dict = 0.00 MB all = 0.0652 GB } UnZip/Zip{ index = 9.4346 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 5.500392 GB avg = 0.277 KB (by entry num) total key len = 0.503409 GB avg = 0.025 KB total ukey len = 0.344509 GB avg = 0.017 KB total ukey num = 0.019862455 Billion total entry num = 0.019862455 Billion write speed all = 88.049667027 MB/sec (with seq num) write speed all = 85.719299809 MB/sec (without seq num) 2024/11/19-15:19:45.159102 139851137550080 [compaction_job.cc:1948] [default] [JOB 116] Generated table #4763: 100309 keys, 65200461 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:45.159158 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000785159133, "cf_name": "default", "job": 116, "event": "table_file_creation", "file_number": 4763, "file_size": 65200461, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 65028256, "index_size": 170112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2407416, "raw_average_key_size": 24, "raw_value_size": 64702036, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 100309, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000784, "oldest_key_time": 0, "file_creation_time": 1732000784, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4763, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:45.175384 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004764.sst) /storage/mytopling/..//tmp/Topling-82038-upTgS4 2024/11/19-15:19:45.235357 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004764.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:19:45.243699 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004764.sst): index pass time = 0.01's, 143.223'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 99079 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:19:45.310118 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004764.sst): first pass time = 0.13's, 492.223'MB/sec 2024/11/19-15:19:45.310175 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004764.sst): old prealloc_size = 73819750, real_size = 64446512 2024/11/19-15:19:45.310244 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004764.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 99079 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0639 GB zip-val = 0.0643 GB avg-val = 645.26 avg-zval = 648.51 2024/11/19-15:19:45.310326 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004764.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.075 sec, Key+Value = 0.066 GB 2024/11/19-15:19:45.381482 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004764.sst): second pass time = 0.00's,155929848.780'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 903.639'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2560.546'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.534'MB/sec write SST data time = 0.02's, 2589.026'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.064'GB zip my value throughput = 155929848.780'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 99079 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.26 avg-zval = 648.51 usrkeys = 99079 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.26 avg-zval = 648.51 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7848 UnZipSize{ index = 0.0016 GB value = 0.0639 GB dict = 0.00 MB all = 0.0663 GB } __ZipSize{ index = 0.0002 GB value = 0.0643 GB dict = 0.00 MB all = 0.0644 GB } UnZip/Zip{ index = 9.4325 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 5.564323 GB avg = 0.279 KB (by entry num) total key len = 0.505787 GB avg = 0.025 KB total ukey len = 0.346094 GB avg = 0.017 KB total ukey num = 0.019961534 Billion total entry num = 0.019961534 Billion write speed all = 88.727030485 MB/sec (with seq num) write speed all = 86.392802371 MB/sec (without seq num) 2024/11/19-15:19:45.386298 139851137550080 [compaction_job.cc:1948] [default] [JOB 116] Generated table #4764: 99079 keys, 64423629 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:45.386350 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000785386326, "cf_name": "default", "job": 116, "event": "table_file_creation", "file_number": 4764, "file_size": 64423629, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64253472, "index_size": 168064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2377896, "raw_average_key_size": 24, "raw_value_size": 63931238, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 99079, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000785, "oldest_key_time": 0, "file_creation_time": 1732000785, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4764, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:45.403113 139851137550080 [compaction_job.cc:703] job-00116: subcompact[0], size: 0.129624 G, files: /storage/mytopling/.rocksdb [004763,004764] 2024/11/19-15:19:45.417171 139851137550080 (Original Log Time 2024/11/19-15:19:44.945198) EVENT_LOG_v1 {"time_micros": 1732000784945181, "job": 116, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4636, 4637, 4638, 4640], "score": 6.37143, "input_data_size": 134026816, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:45.417181 139851137550080 (Original Log Time 2024/11/19-15:19:45.416795) [compaction_job.cc:2022] [default] [JOB 116] Compacted 4@1 files to L2 => 129624090 bytes 2024/11/19-15:19:45.417184 139851137550080 (Original Log Time 2024/11/19-15:19:45.417055) [compaction_job.cc:1224] [default] compacted to: files[2 96 32 16 0 0 0] max score 4.45 (106 files need compaction), MB/sec: 292.7 rd, 283.1 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.8, 0.0 +0.0 blob) out(123.6 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 199388, records dropped: 0 output_compression: Snappy 2024/11/19-15:19:45.417200 139851137550080 (Original Log Time 2024/11/19-15:19:45.417080) EVENT_LOG_v1 {"time_micros": 1732000785417068, "cf": "default", "job": 116, "event": "compaction_finished", "compaction_time_micros": 457928, "compaction_time_cpu_micros": 455838, "output_level": 2, "num_output_files": 2, "total_output_size": 129624090, "num_input_records": 199388, "num_output_records": 199388, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 96, 32, 16, 0, 0, 0]} 2024/11/19-15:19:45.423358 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004592.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:45.423383 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000785423380, "job": 116, "event": "table_file_deletion", "file_number": 4592} 2024/11/19-15:19:45.423505 139851137550080 [compaction_job.cc:2437] [default] [JOB 128] Compacting 2@1 files to L2, score 4.45, subcompactions 1 : 1 2024/11/19-15:19:45.423513 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 126 Base level 1, inputs: [4666(31MB) 4668(14MB)] 2024/11/19-15:19:45.671973 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-00 2024/11/19-15:19:45.672078 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00124/att-00/compact.done) = No such file or directory: 10.661 sec on http probe failed: 5, giveup 2024/11/19-15:19:45.672098 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00124/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 105.246 sec, real 10.958 (submit 288.246 ms wait 10.661 s), input 105.246 MiB, send shutdown 2024/11/19-15:19:45.677528 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-00 2024/11/19-15:19:45.677757 139851148035840 [dcompact_etcd.cc:1773] job-00124/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:45.741545 139851137550080 [dcompact_etcd.cc:1163] job-00128/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.317 sec, input = 46.255 MiB 2024/11/19-15:19:45.802328 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-01 2024/11/19-15:19:45.802395 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00122/att-01/compact.done) = No such file or directory: 5.209 sec on http probe failed: 2, retry 2024/11/19-15:19:45.972966 139851148035840 [dcompact_etcd.cc:1163] job-00124/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.295 sec, input = 105.246 MiB 2024/11/19-15:19:46.390098 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-00 2024/11/19-15:19:46.390180 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00126/att-00/compact.done) = No such file or directory: 5.204 sec on http probe failed: 2, retry 2024/11/19-15:19:46.734297 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-02 2024/11/19-15:19:46.734395 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00117/att-02/compact.done) = No such file or directory: 11.223 sec on http probe failed: 5, giveup 2024/11/19-15:19:46.734414 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00117/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.241 sec, real 11.548 (submit 315.717 ms wait 11.223 s), input 127.241 MiB, send shutdown 2024/11/19-15:19:46.738827 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00117/att-02 2024/11/19-15:19:46.739069 139851169007360 [dcompact_etcd.cc:1773] job-00117/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:46.739300 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00117) = success 2024/11/19-15:19:46.739306 139851169007360 [dcompact_etcd.cc:1773] job-00117/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:46.739324 139851169007360 (Original Log Time 2024/11/19-15:19:11.966311) EVENT_LOG_v1 {"time_micros": 1732000751966305, "job": 117, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4642, 4644, 4647, 4648], "score": 6.12178, "input_data_size": 134027968, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:46.739331 139851169007360 [compaction_job.cc:2437] [default] [JOB 117] Compacting 4@1 files to L2, score 6.12, subcompactions 1 : 1 2024/11/19-15:19:46.739339 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 115 Base level 1, inputs: [4642(31MB) 4644(31MB) 4647(31MB) 4648(31MB)] 2024/11/19-15:19:46.739606 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004765.sst) /storage/mytopling/..//tmp/Topling-82038-0WVNBu 2024/11/19-15:19:46.798769 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004765.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:19:46.807109 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004765.sst): index pass time = 0.01's, 145.098'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 100340 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:19:46.870911 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004765.sst): first pass time = 0.13's, 511.151'MB/sec 2024/11/19-15:19:46.870975 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004765.sst): old prealloc_size = 73819750, real_size = 65222816 2024/11/19-15:19:46.871047 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004765.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 100340 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0650 GB avg-val = 644.82 avg-zval = 648.07 2024/11/19-15:19:46.871124 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004765.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.072 sec, Key+Value = 0.067 GB 2024/11/19-15:19:46.906809 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-00 2024/11/19-15:19:46.906912 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00128/att-00/compact.done) = No such file or directory: 1.156 sec on http probe failed: 1, retry 2024/11/19-15:19:46.940865 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004765.sst): second pass time = 0.00's,155531307.692'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 933.059'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2536.987'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.536'MB/sec write SST data time = 0.02's, 2838.791'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 155531307.692'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 100340 avg-key = 24.00 avg-zkey = 1.70 avg-val = 644.82 avg-zval = 648.07 usrkeys = 100340 avg-key = 16.00 avg-zkey = 1.70 avg-val = 644.82 avg-zval = 648.07 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7942 UnZipSize{ index = 0.0016 GB value = 0.0647 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0002 GB value = 0.0650 GB dict = 0.00 MB all = 0.0652 GB } UnZip/Zip{ index = 9.4340 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9715 } ---------------------------- total value len = 5.629024 GB avg = 0.281 KB (by entry num) total key len = 0.508195 GB avg = 0.025 KB total ukey len = 0.347700 GB avg = 0.017 KB total ukey num = 0.020061874 Billion total entry num = 0.020061874 Billion write speed all = 87.708771004 MB/sec (with seq num) write speed all = 85.415090689 MB/sec (without seq num) 2024/11/19-15:19:46.945660 139851169007360 [compaction_job.cc:1948] [default] [JOB 117] Generated table #4765: 100340 keys, 65199629 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:46.945710 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000786945686, "cf_name": "default", "job": 117, "event": "table_file_creation", "file_number": 4765, "file_size": 65199629, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 65027344, "index_size": 170176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2408160, "raw_average_key_size": 24, "raw_value_size": 64701024, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 100340, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000786, "oldest_key_time": 0, "file_creation_time": 1732000786, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4765, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:46.961435 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004766.sst) /storage/mytopling/..//tmp/Topling-82038-PfoRKL 2024/11/19-15:19:47.020951 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004766.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:19:47.026219 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004766.sst): index pass time = 0.01's, 227.279'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 99118 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:19:47.089486 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004766.sst): first pass time = 0.13's, 517.932'MB/sec 2024/11/19-15:19:47.089536 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004766.sst): old prealloc_size = 73819750, real_size = 64449232 2024/11/19-15:19:47.089617 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004766.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 99118 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0639 GB zip-val = 0.0643 GB avg-val = 645.03 avg-zval = 648.28 2024/11/19-15:19:47.089692 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004766.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.069 sec, Key+Value = 0.066 GB 2024/11/19-15:19:47.158470 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004766.sst): second pass time = 0.00's,113559007.105'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 934.834'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2147.251'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.564'MB/sec write SST data time = 0.02's, 2583.170'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.064'GB zip my value throughput = 113559007.105'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 99118 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.03 avg-zval = 648.28 usrkeys = 99118 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.03 avg-zval = 648.28 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7847 UnZipSize{ index = 0.0016 GB value = 0.0639 GB dict = 0.00 MB all = 0.0663 GB } __ZipSize{ index = 0.0002 GB value = 0.0643 GB dict = 0.00 MB all = 0.0644 GB } UnZip/Zip{ index = 9.4290 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1061 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 5.692958 GB avg = 0.282 KB (by entry num) total key len = 0.510574 GB avg = 0.025 KB total ukey len = 0.349286 GB avg = 0.017 KB total ukey num = 0.020160992 Billion total entry num = 0.020160992 Billion write speed all = 88.381610815 MB/sec (with seq num) write speed all = 86.083744290 MB/sec (without seq num) 2024/11/19-15:19:47.162294 139851169007360 [compaction_job.cc:1948] [default] [JOB 117] Generated table #4766: 99118 keys, 64426317 bytes (need compaction), temperature: kUnknown 2024/11/19-15:19:47.162347 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000787162323, "cf_name": "default", "job": 117, "event": "table_file_creation", "file_number": 4766, "file_size": 64426317, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64256064, "index_size": 168192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2378832, "raw_average_key_size": 24, "raw_value_size": 63933721, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 99118, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000786, "oldest_key_time": 0, "file_creation_time": 1732000786, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4766, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:19:47.177664 139851169007360 [compaction_job.cc:703] job-00117: subcompact[0], size: 0.129626 G, files: /storage/mytopling/.rocksdb [004765,004766] 2024/11/19-15:19:47.178554 139851169007360 (Original Log Time 2024/11/19-15:19:46.739367) EVENT_LOG_v1 {"time_micros": 1732000786739351, "job": 117, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4642, 4644, 4647, 4648], "score": 6.12178, "input_data_size": 134027968, "oldest_snapshot_seqno": -1} 2024/11/19-15:19:47.178565 139851169007360 (Original Log Time 2024/11/19-15:19:47.178167) [compaction_job.cc:2022] [default] [JOB 117] Compacted 4@1 files to L2 => 129625946 bytes 2024/11/19-15:19:47.178568 139851169007360 (Original Log Time 2024/11/19-15:19:47.178425) [compaction_job.cc:1224] [default] compacted to: files[2 92 34 16 0 0 0] max score 4.36 (106 files need compaction), MB/sec: 305.8 rd, 295.7 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.8, 0.0 +0.0 blob) out(123.6 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 199458, records dropped: 0 output_compression: Snappy 2024/11/19-15:19:47.178590 139851169007360 (Original Log Time 2024/11/19-15:19:47.178451) EVENT_LOG_v1 {"time_micros": 1732000787178438, "cf": "default", "job": 117, "event": "compaction_finished", "compaction_time_micros": 438311, "compaction_time_cpu_micros": 436448, "output_level": 2, "num_output_files": 2, "total_output_size": 129625946, "num_input_records": 199458, "num_output_records": 199458, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 92, 34, 16, 0, 0, 0]} 2024/11/19-15:19:47.187051 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004577.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:47.187077 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000787187075, "job": 117, "event": "table_file_deletion", "file_number": 4577} 2024/11/19-15:19:47.195102 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004580.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:47.195124 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000787195121, "job": 117, "event": "table_file_deletion", "file_number": 4580} 2024/11/19-15:19:47.198627 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004581.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:47.198647 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000787198645, "job": 117, "event": "table_file_deletion", "file_number": 4581} 2024/11/19-15:19:47.207192 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004582.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:47.207213 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000787207210, "job": 117, "event": "table_file_deletion", "file_number": 4582} 2024/11/19-15:19:47.207390 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004583.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:47.207401 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000787207399, "job": 117, "event": "table_file_deletion", "file_number": 4583} 2024/11/19-15:19:47.209033 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004584.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:47.209051 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000787209049, "job": 117, "event": "table_file_deletion", "file_number": 4584} 2024/11/19-15:19:47.217230 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004587.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:47.217249 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000787217247, "job": 117, "event": "table_file_deletion", "file_number": 4587} 2024/11/19-15:19:47.224183 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004591.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:47.224202 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000787224200, "job": 117, "event": "table_file_deletion", "file_number": 4591} 2024/11/19-15:19:47.230853 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004598.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:47.230872 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000787230870, "job": 117, "event": "table_file_deletion", "file_number": 4598} 2024/11/19-15:19:47.239259 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004622.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:19:47.239279 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000787239277, "job": 117, "event": "table_file_deletion", "file_number": 4622} 2024/11/19-15:19:47.239442 139851169007360 [compaction_job.cc:2437] [default] [JOB 129] Compacting 4@1 files to L2, score 4.36, subcompactions 1 : 1 2024/11/19-15:19:47.239451 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 127 Base level 1, inputs: [4669(31MB) 4671(31MB) 4672(31MB) 4674(31MB)] 2024/11/19-15:19:47.520306 139851169007360 [dcompact_etcd.cc:1163] job-00129/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 126.889 MiB 2024/11/19-15:19:47.779234 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-00 2024/11/19-15:19:47.779339 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00127/att-00/compact.done) = No such file or directory: 3.195 sec on http probe failed: 1, retry 2024/11/19-15:19:47.809944 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-01 2024/11/19-15:19:47.810001 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00122/att-01/compact.done) = No such file or directory: 7.217 sec on http probe failed: 3, retry 2024/11/19-15:19:47.836616 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-00 2024/11/19-15:19:47.836670 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00128/att-00/compact.done) = No such file or directory: 2.091 sec on http probe failed: 2, retry 2024/11/19-15:19:48.258316 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-01 2024/11/19-15:19:48.258449 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00123/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:19:48.398351 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-00 2024/11/19-15:19:48.398455 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00126/att-00/compact.done) = No such file or directory: 7.214 sec on http probe failed: 3, retry 2024/11/19-15:19:48.612591 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-01 2024/11/19-15:19:48.612669 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00124/att-01/compact.done) = No such file or directory: 2.631 sec on http probe failed: 1, retry 2024/11/19-15:19:48.770100 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-00 2024/11/19-15:19:48.770200 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00128/att-00/compact.done) = No such file or directory: 3.020 sec on http probe failed: 3, retry 2024/11/19-15:19:49.700649 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-00 2024/11/19-15:19:49.700728 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00128/att-00/compact.done) = No such file or directory: 3.954 sec on http probe failed: 4, retry 2024/11/19-15:19:49.786907 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-00 2024/11/19-15:19:49.786974 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00127/att-00/compact.done) = No such file or directory: 5.201 sec on http probe failed: 2, retry 2024/11/19-15:19:49.814573 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-01 2024/11/19-15:19:49.814640 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00122/att-01/compact.done) = No such file or directory: 9.225 sec on http probe failed: 4, retry 2024/11/19-15:19:50.266890 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-01 2024/11/19-15:19:50.266979 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00123/att-01/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:19:50.404454 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-00 2024/11/19-15:19:50.404536 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00126/att-00/compact.done) = No such file or directory: 9.222 sec on http probe failed: 4, retry 2024/11/19-15:19:50.621444 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-01 2024/11/19-15:19:50.621526 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00124/att-01/compact.done) = No such file or directory: 4.640 sec on http probe failed: 2, retry 2024/11/19-15:19:50.636031 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-00 2024/11/19-15:19:50.636092 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00128/att-00/compact.done) = No such file or directory: 4.884 sec on http probe failed: 5, giveup 2024/11/19-15:19:50.636108 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00128/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 46.255 sec, real 5.212 (submit 317.488 ms wait 4.884 s), input 46.255 MiB, send shutdown 2024/11/19-15:19:50.644718 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-00 2024/11/19-15:19:50.644952 139851137550080 [dcompact_etcd.cc:1773] job-00128/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:50.700861 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-00 2024/11/19-15:19:50.700968 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00129/att-00/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:19:50.935327 139851137550080 [dcompact_etcd.cc:1163] job-00128/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.290 sec, input = 46.255 MiB 2024/11/19-15:19:51.578835 139852349699840 [db_impl_write.cc:2181] [default] New memtable created with log file: #4767. Immutable memtables: 0. 2024/11/19-15:19:51.579520 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 130] Syncing log #4758 2024/11/19-15:19:51.579621 139850957195008 (Original Log Time 2024/11/19-15:19:51.579301) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:19:51.579632 139850957195008 [flush_job.cc:879] [default] [JOB 130] Flushing memtable with next log file: 4767 2024/11/19-15:19:51.579652 139850957195008 EVENT_LOG_v1 {"time_micros": 1732000791579647, "job": 130, "event": "flush_started", "num_memtables": 1, "num_entries": 1243677, "num_deletes": 0, "total_data_size": 491644381, "memory_usage": 541064584, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:19:51.579656 139850957195008 [flush_job.cc:910] [default] [JOB 130] Level-0 flush table #4768: started 2024/11/19-15:19:51.579984 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000014.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004768.sst in ConvertToSST, mem_size = 515.999 M, time = 319.939 us 2024/11/19-15:19:51.792272 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-00 2024/11/19-15:19:51.792399 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00127/att-00/compact.done) = No such file or directory: 7.209 sec on http probe failed: 3, retry 2024/11/19-15:19:51.824142 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-01 2024/11/19-15:19:51.824257 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00122/att-01/compact.done) = No such file or directory: 11.229 sec on http probe failed: 5, giveup 2024/11/19-15:19:51.824294 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00122/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.823 sec, real 11.504 (submit 264.737 ms wait 11.229 s), input 127.823 MiB, send shutdown 2024/11/19-15:19:51.833583 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-01 2024/11/19-15:19:51.846588 139851127064320 [dcompact_etcd.cc:1773] job-00122/att-01: CleanFiles 0.013 sec : 0.000 2024/11/19-15:19:52.097004 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-01 2024/11/19-15:19:52.097093 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00128/att-01/compact.done) = No such file or directory: 1.156 sec on http probe failed: 1, retry 2024/11/19-15:19:52.106934 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004768.sst): fsize = 516.001 M, time(ms): open: 0.033, seek: 0.014, finish: 0.036, meta: 0.012, Flush: 0.012, sync: 526.813, close: 0.015, all: 526.935 2024/11/19-15:19:52.108053 139850957195008 [flush_job.cc:1083] [default] [JOB 130] Flush lasted 528433 microseconds, and 423858 cpu microseconds. 2024/11/19-15:19:52.130521 139850957195008 (Original Log Time 2024/11/19-15:19:52.106973) [flush_job.cc:1029] [default] [JOB 130] Level-0 flush table #4768: 541066010 bytes OK (needs compaction) 2024/11/19-15:19:52.130531 139850957195008 (Original Log Time 2024/11/19-15:19:52.108069) [memtable_list.cc:560] [default] Level-0 commit table #4768 started 2024/11/19-15:19:52.130535 139850957195008 (Original Log Time 2024/11/19-15:19:52.130441) [memtable_list.cc:759] [default] Level-0 commit table #4768: memtable #1 done 2024/11/19-15:19:52.130537 139850957195008 (Original Log Time 2024/11/19-15:19:52.130466) EVENT_LOG_v1 {"time_micros": 1732000792130460, "job": 130, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [3, 92, 34, 16, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:19:52.130540 139850957195008 (Original Log Time 2024/11/19-15:19:52.130492) [db_impl_compaction_flush.cc:310] [default] Level summary: files[3 92 34 16 0 0 0] max score 4.11 (103 files need compaction) 2024/11/19-15:19:52.160128 139851127064320 [dcompact_etcd.cc:1163] job-00122/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.313 sec, input = 127.823 MiB 2024/11/19-15:19:52.272842 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-01 2024/11/19-15:19:52.272946 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00123/att-01/compact.done) = No such file or directory: 7.213 sec on http probe failed: 3, retry 2024/11/19-15:19:52.412832 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-00 2024/11/19-15:19:52.412929 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00126/att-00/compact.done) = No such file or directory: 11.228 sec on http probe failed: 5, giveup 2024/11/19-15:19:52.412948 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00126/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.819 sec, real 11.564 (submit 327.205 ms wait 11.228 s), input 127.819 MiB, send shutdown 2024/11/19-15:19:52.418112 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-00 2024/11/19-15:19:52.418356 139851158521600 [dcompact_etcd.cc:1773] job-00126/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:52.626244 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-01 2024/11/19-15:19:52.626336 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00124/att-01/compact.done) = No such file or directory: 6.649 sec on http probe failed: 3, retry 2024/11/19-15:19:52.699538 139851158521600 [dcompact_etcd.cc:1163] job-00126/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 127.819 MiB 2024/11/19-15:19:52.710274 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-00 2024/11/19-15:19:52.710336 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00129/att-00/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:19:53.029514 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-01 2024/11/19-15:19:53.029601 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00128/att-01/compact.done) = No such file or directory: 2.087 sec on http probe failed: 2, retry 2024/11/19-15:19:53.798364 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-00 2024/11/19-15:19:53.798450 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00127/att-00/compact.done) = No such file or directory: 9.214 sec on http probe failed: 4, retry 2024/11/19-15:19:53.961178 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-01 2024/11/19-15:19:53.961256 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00128/att-01/compact.done) = No such file or directory: 3.019 sec on http probe failed: 3, retry 2024/11/19-15:19:54.282290 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-01 2024/11/19-15:19:54.282390 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00123/att-01/compact.done) = No such file or directory: 9.219 sec on http probe failed: 4, retry 2024/11/19-15:19:54.636645 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-01 2024/11/19-15:19:54.636742 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00124/att-01/compact.done) = No such file or directory: 8.653 sec on http probe failed: 4, retry 2024/11/19-15:19:54.718288 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-00 2024/11/19-15:19:54.718372 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00129/att-00/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:19:54.891296 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-01 2024/11/19-15:19:54.891375 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00128/att-01/compact.done) = No such file or directory: 3.951 sec on http probe failed: 4, retry 2024/11/19-15:19:55.365468 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-02 2024/11/19-15:19:55.365555 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00122/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:19:55.803734 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-00 2024/11/19-15:19:55.803808 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00127/att-00/compact.done) = No such file or directory: 11.220 sec on http probe failed: 5, giveup 2024/11/19-15:19:55.803826 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00127/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.816 sec, real 11.536 (submit 311.104 ms wait 11.220 s), input 127.816 MiB, send shutdown 2024/11/19-15:19:55.812450 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-00 2024/11/19-15:19:55.824365 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-01 2024/11/19-15:19:55.824421 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00128/att-01/compact.done) = No such file or directory: 4.881 sec on http probe failed: 5, giveup 2024/11/19-15:19:55.824435 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00128/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 46.255 sec, real 5.179 (submit 290.297 ms wait 4.881 s), input 46.255 MiB, send shutdown 2024/11/19-15:19:55.832446 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-01 2024/11/19-15:19:55.904144 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-01 2024/11/19-15:19:55.904248 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00126/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:19:56.027041 139851179493120 [dcompact_etcd.cc:1773] job-00127/att-00: CleanFiles 0.215 sec : 0.000 2024/11/19-15:19:56.027138 139851137550080 [dcompact_etcd.cc:1773] job-00128/att-01: CleanFiles 0.195 sec : 0.000 2024/11/19-15:19:56.291528 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-01 2024/11/19-15:19:56.291610 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00123/att-01/compact.done) = No such file or directory: 11.229 sec on http probe failed: 5, giveup 2024/11/19-15:19:56.291629 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00123/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.819 sec, real 11.537 (submit 299.358 ms wait 11.229 s), input 127.819 MiB, send shutdown 2024/11/19-15:19:56.299899 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-01 2024/11/19-15:19:56.303073 139851137550080 [dcompact_etcd.cc:1163] job-00128/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.276 sec, input = 46.255 MiB 2024/11/19-15:19:56.319992 139850978166528 [dcompact_etcd.cc:1773] job-00123/att-01: CleanFiles 0.020 sec : 0.000 2024/11/19-15:19:56.323040 139851179493120 [dcompact_etcd.cc:1163] job-00127/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.296 sec, input = 127.816 MiB 2024/11/19-15:19:56.618226 139850978166528 [dcompact_etcd.cc:1163] job-00123/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.298 sec, input = 127.819 MiB 2024/11/19-15:19:56.644672 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-01 2024/11/19-15:19:56.644733 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00124/att-01/compact.done) = No such file or directory: 10.664 sec on http probe failed: 5, giveup 2024/11/19-15:19:56.644749 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00124/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 105.246 sec, real 10.967 (submit 295.083 ms wait 10.664 s), input 105.246 MiB, send shutdown 2024/11/19-15:19:56.649489 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-01 2024/11/19-15:19:56.649720 139851148035840 [dcompact_etcd.cc:1773] job-00124/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:56.730042 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-00 2024/11/19-15:19:56.730106 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00129/att-00/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:19:56.923639 139851148035840 [dcompact_etcd.cc:1163] job-00124/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.274 sec, input = 105.246 MiB 2024/11/19-15:19:57.375881 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-02 2024/11/19-15:19:57.375985 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00122/att-02/compact.done) = No such file or directory: 5.206 sec on http probe failed: 2, retry 2024/11/19-15:19:57.464370 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-02 2024/11/19-15:19:57.464447 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00128/att-02/compact.done) = No such file or directory: 1.156 sec on http probe failed: 1, retry 2024/11/19-15:19:57.913162 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-01 2024/11/19-15:19:57.913283 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00126/att-01/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:19:58.398646 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-02 2024/11/19-15:19:58.398722 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00128/att-02/compact.done) = No such file or directory: 2.087 sec on http probe failed: 2, retry 2024/11/19-15:19:58.739610 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-00 2024/11/19-15:19:58.739708 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00129/att-00/compact.done) = No such file or directory: 11.210 sec on http probe failed: 5, giveup 2024/11/19-15:19:58.739736 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00129/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.889 sec, real 11.500 (submit 280.413 ms wait 11.210 s), input 126.889 MiB, send shutdown 2024/11/19-15:19:58.745101 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-00 2024/11/19-15:19:58.745384 139851169007360 [dcompact_etcd.cc:1773] job-00129/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:19:59.030584 139851169007360 [dcompact_etcd.cc:1163] job-00129/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.285 sec, input = 126.889 MiB 2024/11/19-15:19:59.332416 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-02 2024/11/19-15:19:59.332506 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00128/att-02/compact.done) = No such file or directory: 3.021 sec on http probe failed: 3, retry 2024/11/19-15:19:59.385090 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-02 2024/11/19-15:19:59.385164 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00122/att-02/compact.done) = No such file or directory: 7.216 sec on http probe failed: 3, retry 2024/11/19-15:19:59.524134 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-01 2024/11/19-15:19:59.524210 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00127/att-01/compact.done) = No such file or directory: 3.195 sec on http probe failed: 1, retry 2024/11/19-15:19:59.563699 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-02 2024/11/19-15:19:59.563773 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00124/att-02/compact.done) = No such file or directory: 2.631 sec on http probe failed: 1, retry 2024/11/19-15:19:59.822416 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-02 2024/11/19-15:19:59.822525 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00123/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:19:59.919270 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-01 2024/11/19-15:19:59.919348 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00126/att-01/compact.done) = No such file or directory: 7.214 sec on http probe failed: 3, retry 2024/11/19-15:20:00.264166 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-02 2024/11/19-15:20:00.264266 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00128/att-02/compact.done) = No such file or directory: 3.955 sec on http probe failed: 4, retry 2024/11/19-15:20:01.198437 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-02 2024/11/19-15:20:01.198597 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00128/att-02/compact.done) = No such file or directory: 4.886 sec on http probe failed: 5, giveup 2024/11/19-15:20:01.198630 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00128/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 46.255 sec, real 5.171 (submit 275.828 ms wait 4.886 s), input 46.255 MiB, send shutdown 2024/11/19-15:20:01.208448 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00128/att-02 2024/11/19-15:20:01.208750 139851137550080 [dcompact_etcd.cc:1773] job-00128/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:01.210702 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00128) = success 2024/11/19-15:20:01.210717 139851137550080 [dcompact_etcd.cc:1773] job-00128/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:20:01.210744 139851137550080 (Original Log Time 2024/11/19-15:19:45.423525) EVENT_LOG_v1 {"time_micros": 1732000785423519, "job": 128, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4666, 4668], "score": 4.44536, "input_data_size": 48315462, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:01.210757 139851137550080 [compaction_job.cc:2437] [default] [JOB 128] Compacting 2@1 files to L2, score 4.45, subcompactions 1 : 1 2024/11/19-15:20:01.210768 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 126 Base level 1, inputs: [4666(31MB) 4668(14MB)] 2024/11/19-15:20:01.211056 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004769.sst) /storage/mytopling/..//tmp/Topling-82038-WnaopN 2024/11/19-15:20:01.327942 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004769.sst: sumWaitingMem = 0.004 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0044 GB, waited 0.000 sec, Key+Value = 0.049 GB 2024/11/19-15:20:01.352503 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004769.sst): index pass time = 0.02's, 218.604'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 335080 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0054 GB zip-key = 0.0006 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:20:01.393918 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-02 2024/11/19-15:20:01.394022 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00122/att-02/compact.done) = No such file or directory: 9.225 sec on http probe failed: 4, retry 2024/11/19-15:20:01.402426 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004769.sst): first pass time = 0.19's, 253.472'MB/sec 2024/11/19-15:20:01.402477 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004769.sst): old prealloc_size = 53147008, real_size = 40205744 2024/11/19-15:20:01.402650 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004769.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 335080 prefix = 4 raw-key = 0.0054 GB zip-key = 0.0006 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0391 GB zip-val = 0.0395 GB avg-val = 116.75 avg-zval = 117.84 2024/11/19-15:20:01.402700 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004769.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0008 GB, waited 0.075 sec, Key+Value = 0.049 GB 2024/11/19-15:20:01.472368 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004769.sst): second pass time = 0.00's,125384897.436'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 574.242'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3797.626'MB/sec (index lex order gen) rebuild zvType time = 0.03's, 2.579'MB/sec write SST data time = 0.04's, 1077.771'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.039'GB zip my value throughput = 125384897.436'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 335080 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.75 avg-zval = 117.84 usrkeys = 335080 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.75 avg-zval = 117.84 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 4783 UnZipSize{ index = 0.0067 GB value = 0.0391 GB dict = 0.00 MB all = 0.0485 GB } __ZipSize{ index = 0.0006 GB value = 0.0395 GB dict = 0.00 MB all = 0.0401 GB } UnZip/Zip{ index = 10.5621 value = 0.9907 dict = 0.00 all = 1.2087 } Zip/UnZip{ index = 0.0947 value = 1.0094 dict = 0.00 all = 0.8273 } ---------------------------- total value len = 5.732078 GB avg = 0.280 KB (by entry num) total key len = 0.519956 GB avg = 0.025 KB total ukey len = 0.355987 GB avg = 0.017 KB total ukey num = 0.020496072 Billion total entry num = 0.020496072 Billion write speed all = 73.984893868 MB/sec (with seq num) write speed all = 72.044533684 MB/sec (without seq num) 2024/11/19-15:20:01.474965 139851137550080 [compaction_job.cc:1948] [default] [JOB 128] Generated table #4769: 335080 keys, 40126104 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:01.475014 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000801474991, "cf_name": "default", "job": 128, "event": "table_file_creation", "file_number": 4769, "file_size": 40126104, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 39487264, "index_size": 634496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 9382240, "raw_average_key_size": 28, "raw_value_size": 39120088, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 335080, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000801, "oldest_key_time": 0, "file_creation_time": 1732000801, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4769, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:01.485758 139851137550080 [compaction_job.cc:703] job-00128: subcompact[0], size: 0.040126 G, files: /storage/mytopling/.rocksdb [004769] 2024/11/19-15:20:01.506427 139851137550080 (Original Log Time 2024/11/19-15:20:01.210808) EVENT_LOG_v1 {"time_micros": 1732000801210784, "job": 128, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4666, 4668], "score": 4.44536, "input_data_size": 48315462, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:01.506438 139851137550080 (Original Log Time 2024/11/19-15:20:01.505567) [compaction_job.cc:2022] [default] [JOB 128] Compacted 2@1 files to L2 => 40126104 bytes 2024/11/19-15:20:01.506441 139851137550080 (Original Log Time 2024/11/19-15:20:01.505826) [compaction_job.cc:1224] [default] compacted to: files[3 90 35 16 0 0 0] max score 4.11 (104 files need compaction), MB/sec: 175.7 rd, 145.9 wr, level 2, files in(2, 0) out(1 +0 blob) MB in(46.1, 0.0 +0.0 blob) out(38.3 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 335080, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:01.506457 139851137550080 (Original Log Time 2024/11/19-15:20:01.505851) EVENT_LOG_v1 {"time_micros": 1732000801505840, "cf": "default", "job": 128, "event": "compaction_finished", "compaction_time_micros": 274963, "compaction_time_cpu_micros": 272260, "output_level": 2, "num_output_files": 1, "total_output_size": 40126104, "num_input_records": 335080, "num_output_records": 335080, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 90, 35, 16, 0, 0, 0]} 2024/11/19-15:20:01.506530 139851137550080 [compaction_job.cc:2437] [default] [JOB 131] Compacting 1@1 files to L2, score 4.11, subcompactions 1 : 1 2024/11/19-15:20:01.506538 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 129 Base level 1, inputs: [4675(31MB)] 2024/11/19-15:20:01.532861 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-01 2024/11/19-15:20:01.532947 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00127/att-01/compact.done) = No such file or directory: 5.201 sec on http probe failed: 2, retry 2024/11/19-15:20:01.569899 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-02 2024/11/19-15:20:01.569972 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00124/att-02/compact.done) = No such file or directory: 4.640 sec on http probe failed: 2, retry 2024/11/19-15:20:01.783212 139851137550080 [dcompact_etcd.cc:1163] job-00131/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.276 sec, input = 31.705 MiB 2024/11/19-15:20:01.831350 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-02 2024/11/19-15:20:01.831446 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00123/att-02/compact.done) = No such file or directory: 5.204 sec on http probe failed: 2, retry 2024/11/19-15:20:01.929375 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-01 2024/11/19-15:20:01.929459 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00126/att-01/compact.done) = No such file or directory: 9.220 sec on http probe failed: 4, retry 2024/11/19-15:20:02.207959 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-01 2024/11/19-15:20:02.208062 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00129/att-01/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:20:02.586004 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-00 2024/11/19-15:20:02.586095 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00131/att-00/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:20:03.229432 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-00 2024/11/19-15:20:03.229538 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00131/att-00/compact.done) = No such file or directory: 1.437 sec on http probe failed: 2, retry 2024/11/19-15:20:03.399909 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-02 2024/11/19-15:20:03.399989 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00122/att-02/compact.done) = No such file or directory: 11.234 sec on http probe failed: 5, giveup 2024/11/19-15:20:03.400007 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00122/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.823 sec, real 11.553 (submit 313.355 ms wait 11.234 s), input 127.823 MiB, send shutdown 2024/11/19-15:20:03.408195 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00122/att-02 2024/11/19-15:20:03.408418 139851127064320 [dcompact_etcd.cc:1773] job-00122/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:03.410314 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00122) = success 2024/11/19-15:20:03.410322 139851127064320 [dcompact_etcd.cc:1773] job-00122/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:20:03.410341 139851127064320 (Original Log Time 2024/11/19-15:19:28.781526) EVENT_LOG_v1 {"time_micros": 1732000768781521, "job": 122, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4652, 4655, 4657, 4659], "score": 5.64959, "input_data_size": 133300364, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:03.410348 139851127064320 [compaction_job.cc:2437] [default] [JOB 122] Compacting 4@1 files to L2, score 5.65, subcompactions 1 : 1 2024/11/19-15:20:03.410356 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 120 Base level 1, inputs: [4652(31MB) 4655(31MB) 4657(31MB) 4659(31MB)] 2024/11/19-15:20:03.410588 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004770.sst) /storage/mytopling/..//tmp/Topling-82038-yZSoW5 2024/11/19-15:20:03.541717 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-01 2024/11/19-15:20:03.541827 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00127/att-01/compact.done) = No such file or directory: 7.210 sec on http probe failed: 3, retry 2024/11/19-15:20:03.555680 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004770.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:20:03.574970 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-02 2024/11/19-15:20:03.575067 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00124/att-02/compact.done) = No such file or directory: 6.646 sec on http probe failed: 3, retry 2024/11/19-15:20:03.589957 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004770.sst): index pass time = 0.03's, 216.480'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463352 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:20:03.657977 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004770.sst): first pass time = 0.25's, 271.284'MB/sec 2024/11/19-15:20:03.658030 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004770.sst): old prealloc_size = 73819750, real_size = 55605536 2024/11/19-15:20:03.658261 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004770.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463352 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.83 avg-zval = 117.87 2024/11/19-15:20:03.658339 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004770.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.103 sec, Key+Value = 0.067 GB 2024/11/19-15:20:03.751600 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004770.sst): second pass time = 0.00's,157828431.487'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 593.194'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3689.701'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.626'MB/sec write SST data time = 0.05's, 1127.754'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 157828431.487'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463352 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.83 avg-zval = 117.87 usrkeys = 463352 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.83 avg-zval = 117.87 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6616 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5746 value = 0.9912 dict = 0.00 all = 1.2093 } Zip/UnZip{ index = 0.0946 value = 1.0088 dict = 0.00 all = 0.8269 } ---------------------------- total value len = 5.786213 GB avg = 0.276 KB (by entry num) total key len = 0.532930 GB avg = 0.025 KB total ukey len = 0.365254 GB avg = 0.017 KB total ukey num = 0.020959424 Billion total entry num = 0.020959424 Billion write speed all = 72.815087938 MB/sec (with seq num) write speed all = 70.882974905 MB/sec (without seq num) 2024/11/19-15:20:03.755103 139851127064320 [compaction_job.cc:1948] [default] [JOB 122] Generated table #4770: 463352 keys, 55495000 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:03.755156 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000803755132, "cf_name": "default", "job": 122, "event": "table_file_creation", "file_number": 4770, "file_size": 55495000, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54613136, "index_size": 876352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12973856, "raw_average_key_size": 28, "raw_value_size": 54135152, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463352, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000803, "oldest_key_time": 0, "file_creation_time": 1732000803, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4770, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:03.768737 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004771.sst) /storage/mytopling/..//tmp/Topling-82038-SLNXfK 2024/11/19-15:20:03.839418 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-02 2024/11/19-15:20:03.839493 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00123/att-02/compact.done) = No such file or directory: 7.213 sec on http probe failed: 3, retry 2024/11/19-15:20:03.872283 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-00 2024/11/19-15:20:03.872345 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00131/att-00/compact.done) = No such file or directory: 2.081 sec on http probe failed: 3, retry 2024/11/19-15:20:03.912839 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004771.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:20:03.935553 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-01 2024/11/19-15:20:03.935671 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00126/att-01/compact.done) = No such file or directory: 11.230 sec on http probe failed: 5, giveup 2024/11/19-15:20:03.935698 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00126/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.819 sec, real 11.517 (submit 281.102 ms wait 11.230 s), input 127.819 MiB, send shutdown 2024/11/19-15:20:03.941400 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-01 2024/11/19-15:20:03.941677 139851158521600 [dcompact_etcd.cc:1773] job-00126/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:03.947451 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004771.sst): index pass time = 0.03's, 213.821'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462156 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:20:04.015685 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004771.sst): first pass time = 0.25's, 271.021'MB/sec 2024/11/19-15:20:04.015744 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004771.sst): old prealloc_size = 73819750, real_size = 55464672 2024/11/19-15:20:04.015993 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004771.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462156 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.81 avg-zval = 117.86 2024/11/19-15:20:04.016073 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004771.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.103 sec, Key+Value = 0.067 GB 2024/11/19-15:20:04.109509 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004771.sst): second pass time = 0.00's,111998082.988'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 590.422'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3551.064'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.597'MB/sec write SST data time = 0.05's, 1129.133'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 111998082.988'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462156 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.81 avg-zval = 117.86 usrkeys = 462156 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.81 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6597 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0554 GB } UnZip/Zip{ index = 10.5058 value = 0.9911 dict = 0.00 all = 1.2090 } Zip/UnZip{ index = 0.0952 value = 1.0090 dict = 0.00 all = 0.8271 } ---------------------------- total value len = 5.840196 GB avg = 0.273 KB (by entry num) total key len = 0.545870 GB avg = 0.025 KB total ukey len = 0.374497 GB avg = 0.017 KB total ukey num = 0.021421580 Billion total entry num = 0.021421580 Billion write speed all = 73.284006982 MB/sec (with seq num) write speed all = 71.317401405 MB/sec (without seq num) 2024/11/19-15:20:04.113101 139851127064320 [compaction_job.cc:1948] [default] [JOB 122] Generated table #4771: 462156 keys, 55354456 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:04.113152 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000804113129, "cf_name": "default", "job": 122, "event": "table_file_creation", "file_number": 4771, "file_size": 55354456, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54469136, "index_size": 879808, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12940368, "raw_average_key_size": 28, "raw_value_size": 53983076, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462156, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000803, "oldest_key_time": 0, "file_creation_time": 1732000803, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4771, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:04.126617 139851127064320 [compaction_job.cc:703] job-00122: subcompact[0], size: 0.110849 G, files: /storage/mytopling/.rocksdb [004770,004771] 2024/11/19-15:20:04.127887 139851127064320 (Original Log Time 2024/11/19-15:20:03.410386) EVENT_LOG_v1 {"time_micros": 1732000803410369, "job": 122, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4652, 4655, 4657, 4659], "score": 5.64959, "input_data_size": 133300364, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:04.127897 139851127064320 (Original Log Time 2024/11/19-15:20:04.127508) [compaction_job.cc:2022] [default] [JOB 122] Compacted 4@1 files to L2 => 110849456 bytes 2024/11/19-15:20:04.127900 139851127064320 (Original Log Time 2024/11/19-15:20:04.127767) [compaction_job.cc:1224] [default] compacted to: files[3 86 37 16 0 0 0] max score 4.04 (105 files need compaction), MB/sec: 186.1 rd, 154.8 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925508, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:04.127915 139851127064320 (Original Log Time 2024/11/19-15:20:04.127792) EVENT_LOG_v1 {"time_micros": 1732000804127779, "cf": "default", "job": 122, "event": "compaction_finished", "compaction_time_micros": 716244, "compaction_time_cpu_micros": 713280, "output_level": 2, "num_output_files": 2, "total_output_size": 110849456, "num_input_records": 925508, "num_output_records": 925508, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 86, 37, 16, 0, 0, 0]} 2024/11/19-15:20:04.128552 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004643.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:04.128572 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000804128570, "job": 122, "event": "table_file_deletion", "file_number": 4643} 2024/11/19-15:20:04.131005 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004645.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:04.131026 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000804131024, "job": 122, "event": "table_file_deletion", "file_number": 4645} 2024/11/19-15:20:04.131142 139851127064320 [compaction_job.cc:2437] [default] [JOB 132] Compacting 4@1 files to L2, score 4.04, subcompactions 1 : 1 2024/11/19-15:20:04.131151 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 130 Base level 1, inputs: [4676(31MB) 4677(31MB) 4678(31MB) 4679(31MB)] 2024/11/19-15:20:04.216963 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-01 2024/11/19-15:20:04.217051 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00129/att-01/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:20:04.225696 139851158521600 [dcompact_etcd.cc:1163] job-00126/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.284 sec, input = 127.819 MiB 2024/11/19-15:20:04.430029 139851127064320 [dcompact_etcd.cc:1163] job-00132/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.298 sec, input = 126.906 MiB 2024/11/19-15:20:04.511522 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-00 2024/11/19-15:20:04.511628 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00131/att-00/compact.done) = No such file or directory: 2.723 sec on http probe failed: 4, retry 2024/11/19-15:20:05.150659 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-00 2024/11/19-15:20:05.150753 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00131/att-00/compact.done) = No such file or directory: 3.363 sec on http probe failed: 5, giveup 2024/11/19-15:20:05.150778 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00131/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.705 sec, real 3.644 (submit 276.173 ms wait 3.363 s), input 31.705 MiB, send shutdown 2024/11/19-15:20:05.157987 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-00 2024/11/19-15:20:05.158259 139851137550080 [dcompact_etcd.cc:1773] job-00131/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:05.449618 139851137550080 [dcompact_etcd.cc:1163] job-00131/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.291 sec, input = 31.705 MiB 2024/11/19-15:20:05.550894 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-01 2024/11/19-15:20:05.550991 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00127/att-01/compact.done) = No such file or directory: 9.219 sec on http probe failed: 4, retry 2024/11/19-15:20:05.585625 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-02 2024/11/19-15:20:05.585742 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00124/att-02/compact.done) = No such file or directory: 8.652 sec on http probe failed: 4, retry 2024/11/19-15:20:05.844555 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-02 2024/11/19-15:20:05.844633 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00123/att-02/compact.done) = No such file or directory: 9.221 sec on http probe failed: 4, retry 2024/11/19-15:20:06.227529 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-01 2024/11/19-15:20:06.227616 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00129/att-01/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:20:06.247151 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-01 2024/11/19-15:20:06.247223 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00131/att-01/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:20:06.890938 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-01 2024/11/19-15:20:06.891034 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00131/att-01/compact.done) = No such file or directory: 1.432 sec on http probe failed: 2, retry 2024/11/19-15:20:07.430527 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-02 2024/11/19-15:20:07.430614 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00126/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:20:07.533538 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-01 2024/11/19-15:20:07.533621 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00131/att-01/compact.done) = No such file or directory: 2.076 sec on http probe failed: 3, retry 2024/11/19-15:20:07.558553 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-01 2024/11/19-15:20:07.558614 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00127/att-01/compact.done) = No such file or directory: 11.228 sec on http probe failed: 5, giveup 2024/11/19-15:20:07.558631 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00127/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.816 sec, real 11.531 (submit 295.861 ms wait 11.228 s), input 127.816 MiB, send shutdown 2024/11/19-15:20:07.566331 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-01 2024/11/19-15:20:07.566547 139851179493120 [dcompact_etcd.cc:1773] job-00127/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:07.590431 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-02 2024/11/19-15:20:07.590490 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00124/att-02/compact.done) = No such file or directory: 10.662 sec on http probe failed: 5, giveup 2024/11/19-15:20:07.590506 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00124/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 105.246 sec, real 10.941 (submit 273.834 ms wait 10.662 s), input 105.246 MiB, send shutdown 2024/11/19-15:20:07.599564 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00124/att-02 2024/11/19-15:20:07.599757 139851148035840 [dcompact_etcd.cc:1773] job-00124/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:07.601933 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00124) = success 2024/11/19-15:20:07.601941 139851148035840 [dcompact_etcd.cc:1773] job-00124/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:20:07.601960 139851148035840 (Original Log Time 2024/11/19-15:19:34.713248) EVENT_LOG_v1 {"time_micros": 1732000774713242, "job": 124, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4670, 4673, 4633, 4639], "score": 5.15029, "input_data_size": 109854921, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:07.601967 139851148035840 [compaction_job.cc:2437] [default] [JOB 124] Compacting 4@1 files to L2, score 5.15, subcompactions 1 : 1 2024/11/19-15:20:07.601975 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 122 Base level 1, inputs: [4670(31MB) 4673(9648KB) 4633(31MB) 4639(31MB)] 2024/11/19-15:20:07.602256 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004772.sst) /storage/mytopling/..//tmp/Topling-82038-CZAf5J 2024/11/19-15:20:07.608156 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-00 2024/11/19-15:20:07.608246 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00132/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:20:07.751189 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004772.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:20:07.794724 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004772.sst): index pass time = 0.04's, 170.545'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463573 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:20:07.840660 139851179493120 [dcompact_etcd.cc:1163] job-00127/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.274 sec, input = 127.816 MiB 2024/11/19-15:20:07.852415 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004772.sst): first pass time = 0.25's, 268.279'MB/sec 2024/11/19-15:20:07.852465 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004772.sst): old prealloc_size = 73819750, real_size = 55624432 2024/11/19-15:20:07.852699 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004772.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463573 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.76 avg-zval = 117.85 2024/11/19-15:20:07.852785 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004772.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.102 sec, Key+Value = 0.067 GB 2024/11/19-15:20:07.853471 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-02 2024/11/19-15:20:07.853537 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00123/att-02/compact.done) = No such file or directory: 11.226 sec on http probe failed: 5, giveup 2024/11/19-15:20:07.853555 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00123/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.819 sec, real 11.533 (submit 298.129 ms wait 11.226 s), input 127.819 MiB, send shutdown 2024/11/19-15:20:07.861299 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00123/att-02 2024/11/19-15:20:07.861517 139850978166528 [dcompact_etcd.cc:1773] job-00123/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:07.863448 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00123) = success 2024/11/19-15:20:07.863457 139850978166528 [dcompact_etcd.cc:1773] job-00123/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:20:07.863479 139850978166528 (Original Log Time 2024/11/19-15:19:33.222158) EVENT_LOG_v1 {"time_micros": 1732000773222152, "job": 123, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4661, 4663, 4665, 4667], "score": 5.39994, "input_data_size": 133300236, "oldest_snapshot_seqno": 124640232} 2024/11/19-15:20:07.863486 139850978166528 [compaction_job.cc:2437] [default] [JOB 123] Compacting 4@1 files to L2, score 5.40, subcompactions 1 : 1 2024/11/19-15:20:07.863496 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 121 Base level 1, inputs: [4661(31MB) 4663(31MB) 4665(31MB) 4667(31MB)] 2024/11/19-15:20:07.863746 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004773.sst) /storage/mytopling/..//tmp/Topling-82038-dgErtU 2024/11/19-15:20:07.957712 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004772.sst): second pass time = 0.00's,128267654.028'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.11's, 527.550'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3806.066'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 2.132'MB/sec write SST data time = 0.05's, 1096.299'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 128267654.028'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463573 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.76 avg-zval = 117.85 usrkeys = 463573 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.76 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6615 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5696 value = 0.9908 dict = 0.00 all = 1.2089 } Zip/UnZip{ index = 0.0946 value = 1.0093 dict = 0.00 all = 0.8272 } ---------------------------- total value len = 5.894325 GB avg = 0.269 KB (by entry num) total key len = 0.558850 GB avg = 0.026 KB total ukey len = 0.383769 GB avg = 0.018 KB total ukey num = 0.021885153 Billion total entry num = 0.021885153 Billion write speed all = 70.922167585 MB/sec (with seq num) write speed all = 68.997976903 MB/sec (without seq num) 2024/11/19-15:20:07.968184 139851148035840 [compaction_job.cc:1948] [default] [JOB 124] Generated table #4772: 463573 keys, 55513880 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:07.968237 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000807968214, "cf_name": "default", "job": 124, "event": "table_file_creation", "file_number": 4772, "file_size": 55513880, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54631200, "index_size": 877184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12980044, "raw_average_key_size": 28, "raw_value_size": 54128950, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463573, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000807, "oldest_key_time": 0, "file_creation_time": 1732000807, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4772, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:07.982783 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004774.sst) /storage/mytopling/..//tmp/Topling-82038-GZ2dRV 2024/11/19-15:20:08.043623 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004773.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:20:08.075487 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004774.sst: sumWaitingMem = 0.004 GB, sumWorkingMem = 0.006 GB, nltTrie workingMem = 0.0039 GB, waited 0.000 sec, Key+Value = 0.043 GB 2024/11/19-15:20:08.098491 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004773.sst): index pass time = 0.05's, 135.265'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463515 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:20:08.111473 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004774.sst): index pass time = 0.04's, 132.972'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 298703 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0048 GB zip-key = 0.0006 GB avg-key = 16.00 avg-zkey = 1.91 2024/11/19-15:20:08.144186 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004774.sst): first pass time = 0.16's, 267.998'MB/sec 2024/11/19-15:20:08.144249 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004774.sst): old prealloc_size = 73819750, real_size = 35847664 2024/11/19-15:20:08.144422 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004774.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 298703 prefix = 4 raw-key = 0.0048 GB zip-key = 0.0006 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0349 GB zip-val = 0.0352 GB avg-val = 116.79 avg-zval = 117.86 2024/11/19-15:20:08.144486 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004774.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0007 GB, waited 0.069 sec, Key+Value = 0.043 GB 2024/11/19-15:20:08.156998 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004773.sst): first pass time = 0.29's, 228.854'MB/sec 2024/11/19-15:20:08.157047 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004773.sst): old prealloc_size = 73819750, real_size = 55619696 2024/11/19-15:20:08.157288 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004773.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463515 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.78 avg-zval = 117.85 2024/11/19-15:20:08.157344 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004773.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.114 sec, Key+Value = 0.067 GB 2024/11/19-15:20:08.176861 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-01 2024/11/19-15:20:08.176964 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00131/att-01/compact.done) = No such file or directory: 2.718 sec on http probe failed: 4, retry 2024/11/19-15:20:08.207519 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004774.sst): second pass time = 0.00's,40237524.798'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 565.547'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3353.298'MB/sec (index lex order gen) rebuild zvType time = 0.03's, 2.503'MB/sec write SST data time = 0.03's, 1075.681'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.035'GB zip my value throughput = 40237524.798'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 298703 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.79 avg-zval = 117.86 usrkeys = 298703 avg-key = 20.00 avg-zkey = 1.91 avg-val = 116.79 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 4266 UnZipSize{ index = 0.0060 GB value = 0.0349 GB dict = 0.00 MB all = 0.0432 GB } __ZipSize{ index = 0.0006 GB value = 0.0352 GB dict = 0.00 MB all = 0.0358 GB } UnZip/Zip{ index = 10.4964 value = 0.9910 dict = 0.00 all = 1.2089 } Zip/UnZip{ index = 0.0953 value = 1.0091 dict = 0.00 all = 0.8272 } ---------------------------- total value len = 5.929211 GB avg = 0.267 KB (by entry num) total key len = 0.567214 GB avg = 0.026 KB total ukey len = 0.389743 GB avg = 0.018 KB total ukey num = 0.022183856 Billion total entry num = 0.022183856 Billion write speed all = 71.202009197 MB/sec (with seq num) write speed all = 69.256896158 MB/sec (without seq num) 2024/11/19-15:20:08.219066 139851148035840 [compaction_job.cc:1948] [default] [JOB 124] Generated table #4774: 298703 keys, 35776856 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:08.219134 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000808219101, "cf_name": "default", "job": 124, "event": "table_file_creation", "file_number": 4774, "file_size": 35776856, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 35203680, "index_size": 569152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 8363684, "raw_average_key_size": 28, "raw_value_size": 34885934, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 298703, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000807, "oldest_key_time": 0, "file_creation_time": 1732000807, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4774, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:08.227864 139851148035840 [compaction_job.cc:703] job-00124: subcompact[0], size: 0.091291 G, files: /storage/mytopling/.rocksdb [004772,004774] 2024/11/19-15:20:08.229070 139851148035840 (Original Log Time 2024/11/19-15:20:07.602008) EVENT_LOG_v1 {"time_micros": 1732000807601988, "job": 124, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4670, 4673, 4633, 4639], "score": 5.15029, "input_data_size": 109854921, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:08.229080 139851148035840 (Original Log Time 2024/11/19-15:20:08.228672) [compaction_job.cc:2022] [default] [JOB 124] Compacted 4@1 files to L2 => 91290736 bytes 2024/11/19-15:20:08.229084 139851148035840 (Original Log Time 2024/11/19-15:20:08.228961) [compaction_job.cc:1224] [default] compacted to: files[3 82 39 16 0 0 0] max score 3.79 (103 files need compaction), MB/sec: 175.5 rd, 145.9 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(104.8, 0.0 +0.0 blob) out(87.1 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 762276, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:08.229101 139851148035840 (Original Log Time 2024/11/19-15:20:08.228986) EVENT_LOG_v1 {"time_micros": 1732000808228973, "cf": "default", "job": 124, "event": "compaction_finished", "compaction_time_micros": 625870, "compaction_time_cpu_micros": 604198, "output_level": 2, "num_output_files": 2, "total_output_size": 91290736, "num_input_records": 762276, "num_output_records": 762276, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 82, 39, 16, 0, 0, 0]} 2024/11/19-15:20:08.229168 139851148035840 [compaction_job.cc:2437] [default] [JOB 133] Compacting 4@1 files to L2, score 3.79, subcompactions 1 : 1 2024/11/19-15:20:08.229184 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 131 Base level 1, inputs: [4680(31MB) 4681(31MB) 4682(31MB) 4683(31MB)] 2024/11/19-15:20:08.236943 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-01 2024/11/19-15:20:08.237041 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00129/att-01/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:20:08.255640 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004773.sst): second pass time = 0.00's,163536404.834'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 563.078'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3689.181'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 2.453'MB/sec write SST data time = 0.05's, 1086.328'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 163536404.834'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463515 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 117.85 usrkeys = 463515 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6615 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5683 value = 0.9909 dict = 0.00 all = 1.2090 } Zip/UnZip{ index = 0.0946 value = 1.0092 dict = 0.00 all = 0.8271 } ---------------------------- total value len = 5.983342 GB avg = 0.264 KB (by entry num) total key len = 0.580192 GB avg = 0.026 KB total ukey len = 0.399013 GB avg = 0.018 KB total ukey num = 0.022647371 Billion total entry num = 0.022647371 Billion write speed all = 71.899611743 MB/sec (with seq num) write speed all = 69.914903880 MB/sec (without seq num) 2024/11/19-15:20:08.259086 139850978166528 [compaction_job.cc:1948] [default] [JOB 123] Generated table #4773: 463515 keys, 55509144 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:08.259128 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000808259107, "cf_name": "default", "job": 123, "event": "table_file_creation", "file_number": 4773, "file_size": 55509144, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54626464, "index_size": 877184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12978420, "raw_average_key_size": 28, "raw_value_size": 54130550, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463515, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000807, "oldest_key_time": 0, "file_creation_time": 1732000807, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4773, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:08.272343 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004775.sst) /storage/mytopling/..//tmp/Topling-82038-AYU72Q 2024/11/19-15:20:08.416918 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004775.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:20:08.464872 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004775.sst): index pass time = 0.05's, 154.384'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462043 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:20:08.510942 139851148035840 [dcompact_etcd.cc:1163] job-00133/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 126.877 MiB 2024/11/19-15:20:08.518052 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004775.sst): first pass time = 0.25's, 272.366'MB/sec 2024/11/19-15:20:08.518100 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004775.sst): old prealloc_size = 73819750, real_size = 55454192 2024/11/19-15:20:08.518342 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004775.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462043 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.83 avg-zval = 117.86 2024/11/19-15:20:08.518400 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004775.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.101 sec, Key+Value = 0.067 GB 2024/11/19-15:20:08.611310 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004775.sst): second pass time = 0.00's,62988749.125'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 593.862'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3741.477'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.618'MB/sec write SST data time = 0.05's, 1133.233'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 62988749.125'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462043 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.83 avg-zval = 117.86 usrkeys = 462043 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.83 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6597 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0553 GB } UnZip/Zip{ index = 10.5025 value = 0.9912 dict = 0.00 all = 1.2091 } Zip/UnZip{ index = 0.0952 value = 1.0088 dict = 0.00 all = 0.8270 } ---------------------------- total value len = 6.037323 GB avg = 0.261 KB (by entry num) total key len = 0.593129 GB avg = 0.026 KB total ukey len = 0.408254 GB avg = 0.018 KB total ukey num = 0.023109414 Billion total entry num = 0.023109414 Billion write speed all = 72.350779696 MB/sec (with seq num) write speed all = 70.333440317 MB/sec (without seq num) 2024/11/19-15:20:08.614877 139850978166528 [compaction_job.cc:1948] [default] [JOB 123] Generated table #4775: 462043 keys, 55344024 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:08.614944 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000808614907, "cf_name": "default", "job": 123, "event": "table_file_creation", "file_number": 4775, "file_size": 55344024, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54458656, "index_size": 879872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12937204, "raw_average_key_size": 28, "raw_value_size": 53981358, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462043, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000808, "oldest_key_time": 0, "file_creation_time": 1732000808, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4775, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:08.628548 139850978166528 [compaction_job.cc:703] job-00123: subcompact[0], size: 0.110853 G, files: /storage/mytopling/.rocksdb [004773,004775] 2024/11/19-15:20:08.629953 139850978166528 (Original Log Time 2024/11/19-15:20:07.863528) EVENT_LOG_v1 {"time_micros": 1732000807863507, "job": 123, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4661, 4663, 4665, 4667], "score": 5.39994, "input_data_size": 133300236, "oldest_snapshot_seqno": 124640232} 2024/11/19-15:20:08.629965 139850978166528 (Original Log Time 2024/11/19-15:20:08.629550) [compaction_job.cc:2022] [default] [JOB 123] Compacted 4@1 files to L2 => 110853168 bytes 2024/11/19-15:20:08.629969 139850978166528 (Original Log Time 2024/11/19-15:20:08.629820) [compaction_job.cc:1224] [default] compacted to: files[3 78 41 16 0 0 0] max score 3.54 (101 files need compaction), MB/sec: 174.2 rd, 144.9 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925558, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:08.629983 139850978166528 (Original Log Time 2024/11/19-15:20:08.629847) EVENT_LOG_v1 {"time_micros": 1732000808629834, "cf": "default", "job": 123, "event": "compaction_finished", "compaction_time_micros": 765035, "compaction_time_cpu_micros": 758469, "output_level": 2, "num_output_files": 2, "total_output_size": 110853168, "num_input_records": 925558, "num_output_records": 925558, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 78, 41, 16, 0, 0, 0]} 2024/11/19-15:20:08.633747 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004632.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:08.633775 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000808633771, "job": 123, "event": "table_file_deletion", "file_number": 4632} 2024/11/19-15:20:08.634187 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004641.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:08.634201 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000808634200, "job": 123, "event": "table_file_deletion", "file_number": 4641} 2024/11/19-15:20:08.638094 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004650.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:08.638114 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000808638112, "job": 123, "event": "table_file_deletion", "file_number": 4650} 2024/11/19-15:20:08.641064 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004651.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:08.641086 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000808641084, "job": 123, "event": "table_file_deletion", "file_number": 4651} 2024/11/19-15:20:08.641605 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004653.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:08.641624 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000808641622, "job": 123, "event": "table_file_deletion", "file_number": 4653} 2024/11/19-15:20:08.641770 139850978166528 [compaction_job.cc:2437] [default] [JOB 134] Compacting 2@1 files to L2, score 3.54, subcompactions 1 : 1 2024/11/19-15:20:08.641779 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 132 Base level 1, inputs: [4684(31MB) 4685(31MB)] 2024/11/19-15:20:08.821106 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-01 2024/11/19-15:20:08.821189 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00131/att-01/compact.done) = No such file or directory: 3.362 sec on http probe failed: 5, giveup 2024/11/19-15:20:08.821209 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00131/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.705 sec, real 3.663 (submit 291.244 ms wait 3.362 s), input 31.705 MiB, send shutdown 2024/11/19-15:20:08.828521 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-01 2024/11/19-15:20:08.828738 139851137550080 [dcompact_etcd.cc:1773] job-00131/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:08.936566 139850978166528 [dcompact_etcd.cc:1163] job-00134/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.294 sec, input = 63.407 MiB 2024/11/19-15:20:09.127631 139851137550080 [dcompact_etcd.cc:1163] job-00131/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.299 sec, input = 31.705 MiB 2024/11/19-15:20:09.439544 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-02 2024/11/19-15:20:09.439648 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00126/att-02/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:20:09.617182 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-00 2024/11/19-15:20:09.617309 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00132/att-00/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:20:09.926294 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-02 2024/11/19-15:20:09.926378 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00131/att-02/compact.done) = No such file or directory: 0.793 sec on http probe failed: 1, retry 2024/11/19-15:20:10.246382 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-01 2024/11/19-15:20:10.246481 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00129/att-01/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:20:10.246500 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00129/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.889 sec, real 11.501 (submit 285.092 ms wait 11.207 s), input 126.889 MiB, send shutdown 2024/11/19-15:20:10.256093 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-01 2024/11/19-15:20:10.256356 139851169007360 [dcompact_etcd.cc:1773] job-00129/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:10.537767 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-00 2024/11/19-15:20:10.537868 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00134/att-00/compact.done) = No such file or directory: 1.585 sec on http probe failed: 1, retry 2024/11/19-15:20:10.539147 139851169007360 [dcompact_etcd.cc:1163] job-00129/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.283 sec, input = 126.889 MiB 2024/11/19-15:20:10.568904 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-02 2024/11/19-15:20:10.568980 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00131/att-02/compact.done) = No such file or directory: 1.433 sec on http probe failed: 2, retry 2024/11/19-15:20:10.778813 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4776. Immutable memtables: 0. 2024/11/19-15:20:10.779484 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 135] Syncing log #4767 2024/11/19-15:20:10.779575 139850967680768 (Original Log Time 2024/11/19-15:20:10.779260) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:20:10.779582 139850967680768 [flush_job.cc:879] [default] [JOB 135] Flushing memtable with next log file: 4776 2024/11/19-15:20:10.779603 139850967680768 EVENT_LOG_v1 {"time_micros": 1732000810779598, "job": 135, "event": "flush_started", "num_memtables": 1, "num_entries": 1623100, "num_deletes": 0, "total_data_size": 477235080, "memory_usage": 547093896, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:20:10.779608 139850967680768 [flush_job.cc:910] [default] [JOB 135] Level-0 flush table #4777: started 2024/11/19-15:20:10.780050 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000015.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004777.sst in ConvertToSST, mem_size = 521.749 M, time = 433.914 us 2024/11/19-15:20:11.046177 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-02 2024/11/19-15:20:11.046336 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00127/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:20:11.212338 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-02 2024/11/19-15:20:11.212461 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00131/att-02/compact.done) = No such file or directory: 2.076 sec on http probe failed: 3, retry 2024/11/19-15:20:11.323209 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004777.sst): fsize = 521.751 M, time(ms): open: 0.034, seek: 0.016, finish: 0.048, meta: 0.012, Flush: 0.013, sync: 543.004, close: 0.011, all: 543.138 2024/11/19-15:20:11.324326 139850967680768 [flush_job.cc:1083] [default] [JOB 135] Flush lasted 544749 microseconds, and 372649 cpu microseconds. 2024/11/19-15:20:11.345759 139850967680768 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:20:11.345815 139850967680768 (Original Log Time 2024/11/19-15:20:11.323259) [flush_job.cc:1029] [default] [JOB 135] Level-0 flush table #4777: 547095322 bytes OK (needs compaction) 2024/11/19-15:20:11.345821 139850967680768 (Original Log Time 2024/11/19-15:20:11.324341) [memtable_list.cc:560] [default] Level-0 commit table #4777 started 2024/11/19-15:20:11.345824 139850967680768 (Original Log Time 2024/11/19-15:20:11.345728) [memtable_list.cc:759] [default] Level-0 commit table #4777: memtable #1 done 2024/11/19-15:20:11.345827 139850967680768 (Original Log Time 2024/11/19-15:20:11.345754) EVENT_LOG_v1 {"time_micros": 1732000811345747, "job": 135, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [4, 78, 41, 16, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:20:11.345830 139850967680768 (Original Log Time 2024/11/19-15:20:11.345788) [db_impl_compaction_flush.cc:310] [default] Level summary: files[4 78 41 16 0 0 0] max score 3.42 (100 files need compaction) 2024/11/19-15:20:11.447973 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-02 2024/11/19-15:20:11.448046 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00126/att-02/compact.done) = No such file or directory: 7.214 sec on http probe failed: 3, retry 2024/11/19-15:20:11.625231 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-00 2024/11/19-15:20:11.625354 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00132/att-00/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:20:11.688282 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-00 2024/11/19-15:20:11.688393 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00133/att-00/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:20:11.815067 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-00 2024/11/19-15:20:11.815149 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00134/att-00/compact.done) = No such file or directory: 2.870 sec on http probe failed: 2, retry 2024/11/19-15:20:11.851475 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-02 2024/11/19-15:20:11.851532 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00131/att-02/compact.done) = No such file or directory: 2.719 sec on http probe failed: 4, retry 2024/11/19-15:20:12.492432 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-02 2024/11/19-15:20:12.492510 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00131/att-02/compact.done) = No such file or directory: 3.358 sec on http probe failed: 5, giveup 2024/11/19-15:20:12.492528 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00131/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 31.705 sec, real 3.664 (submit 298.809 ms wait 3.358 s), input 31.705 MiB, send shutdown 2024/11/19-15:20:12.497020 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00131/att-02 2024/11/19-15:20:12.497236 139851137550080 [dcompact_etcd.cc:1773] job-00131/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:12.499109 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00131) = success 2024/11/19-15:20:12.499117 139851137550080 [dcompact_etcd.cc:1773] job-00131/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:20:12.499135 139851137550080 (Original Log Time 2024/11/19-15:20:01.506551) EVENT_LOG_v1 {"time_micros": 1732000801506545, "job": 131, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4675], "score": 4.10524, "input_data_size": 33525202, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:12.499143 139851137550080 [compaction_job.cc:2437] [default] [JOB 131] Compacting 1@1 files to L2, score 4.11, subcompactions 1 : 1 2024/11/19-15:20:12.499151 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 129 Base level 1, inputs: [4675(31MB)] 2024/11/19-15:20:12.499400 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004778.sst) /storage/mytopling/..//tmp/Topling-82038-baR34I 2024/11/19-15:20:12.525871 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004778.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0002 GB, waited 0.000 sec, Key+Value = 0.033 GB 2024/11/19-15:20:12.527626 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004778.sst): index pass time = 0.00's, 148.484'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 32061 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0003 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.73 2024/11/19-15:20:12.558462 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004778.sst): first pass time = 0.06's, 562.988'MB/sec 2024/11/19-15:20:12.558507 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004778.sst): old prealloc_size = 36877722, real_size = 32767472 2024/11/19-15:20:12.558548 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004778.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 32061 prefix = 4 raw-key = 0.0003 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.73 raw-val = 0.0326 GB zip-val = 0.0327 GB avg-val =1016.92 avg-zval =1020.05 2024/11/19-15:20:12.558611 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004778.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.033 sec, Key+Value = 0.033 GB 2024/11/19-15:20:12.588447 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004778.sst): second pass time = 0.00's,108678303.333'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.03's, 1094.871'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1287.172'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 0.377'MB/sec write SST data time = 0.01's, 3790.190'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.033'GB zip my value throughput = 108678303.333'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 32061 avg-key = 20.00 avg-zkey = 1.73 avg-val = 1016.92 avg-zval = 1020.05 usrkeys = 32061 avg-key = 12.00 avg-zkey = 1.73 avg-val = 1016.92 avg-zval = 1020.05 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 3997 UnZipSize{ index = 0.0004 GB value = 0.0326 GB dict = 0.00 MB all = 0.0332 GB } __ZipSize{ index = 0.0001 GB value = 0.0327 GB dict = 0.00 MB all = 0.0328 GB } UnZip/Zip{ index = 6.9496 value = 0.9969 dict = 0.00 all = 1.0148 } Zip/UnZip{ index = 0.1439 value = 1.0031 dict = 0.00 all = 0.9854 } ---------------------------- total value len = 6.069927 GB avg = 0.262 KB (by entry num) total key len = 0.593771 GB avg = 0.026 KB total ukey len = 0.408639 GB avg = 0.018 KB total ukey num = 0.023141475 Billion total entry num = 0.023141475 Billion write speed all = 69.689166441 MB/sec (with seq num) write speed all = 67.753051942 MB/sec (without seq num) 2024/11/19-15:20:12.590408 139851137550080 [compaction_job.cc:1948] [default] [JOB 131] Generated table #4778: 32061 keys, 32760708 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:12.590461 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000812590437, "cf_name": "default", "job": 131, "event": "table_file_creation", "file_number": 4778, "file_size": 32760708, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32703904, "index_size": 55360, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641220, "raw_average_key_size": 20, "raw_value_size": 32603491, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32061, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000812, "oldest_key_time": 0, "file_creation_time": 1732000812, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4778, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:12.597619 139851137550080 [compaction_job.cc:703] job-00131: subcompact[0], size: 0.032761 G, files: /storage/mytopling/.rocksdb [004778] 2024/11/19-15:20:12.598185 139851137550080 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:20:12.598291 139851137550080 (Original Log Time 2024/11/19-15:20:12.499181) EVENT_LOG_v1 {"time_micros": 1732000812499162, "job": 131, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4675], "score": 4.10524, "input_data_size": 33525202, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:12.598297 139851137550080 (Original Log Time 2024/11/19-15:20:12.597882) [compaction_job.cc:2022] [default] [JOB 131] Compacted 1@1 files to L2 => 32760708 bytes 2024/11/19-15:20:12.598300 139851137550080 (Original Log Time 2024/11/19-15:20:12.598154) [compaction_job.cc:1224] [default] compacted to: files[4 77 42 16 0 0 0] max score 3.42 (101 files need compaction), MB/sec: 340.5 rd, 332.8 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(32.0, 0.0 +0.0 blob) out(31.2 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 32061, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:12.598317 139851137550080 (Original Log Time 2024/11/19-15:20:12.598178) EVENT_LOG_v1 {"time_micros": 1732000812598166, "cf": "default", "job": 131, "event": "compaction_finished", "compaction_time_micros": 98449, "compaction_time_cpu_micros": 97902, "output_level": 2, "num_output_files": 1, "total_output_size": 32760708, "num_input_records": 32061, "num_output_records": 32061, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 77, 42, 16, 0, 0, 0]} 2024/11/19-15:20:12.598383 139851137550080 [compaction_job.cc:2437] [default] [JOB 136] Compacting 4@1 files to L2, score 3.42, subcompactions 1 : 1 2024/11/19-15:20:12.598391 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 134 Base level 1, inputs: [4686(31MB) 4688(31MB) 4689(31MB) 4690(31MB)] 2024/11/19-15:20:12.887277 139851137550080 [dcompact_etcd.cc:1163] job-00136/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.288 sec, input = 126.987 MiB 2024/11/19-15:20:13.054630 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-02 2024/11/19-15:20:13.054733 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00127/att-02/compact.done) = No such file or directory: 5.206 sec on http probe failed: 2, retry 2024/11/19-15:20:13.092507 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-00 2024/11/19-15:20:13.092585 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00134/att-00/compact.done) = No such file or directory: 4.147 sec on http probe failed: 3, retry 2024/11/19-15:20:13.456369 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-02 2024/11/19-15:20:13.456630 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00126/att-02/compact.done) = No such file or directory: 9.222 sec on http probe failed: 4, retry 2024/11/19-15:20:13.631164 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-00 2024/11/19-15:20:13.631244 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00132/att-00/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:20:13.697398 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-00 2024/11/19-15:20:13.697459 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00133/att-00/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:20:13.720407 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-02 2024/11/19-15:20:13.720478 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00129/att-02/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:20:14.370029 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-00 2024/11/19-15:20:14.370119 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00134/att-00/compact.done) = No such file or directory: 5.424 sec on http probe failed: 4, retry 2024/11/19-15:20:15.060810 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-02 2024/11/19-15:20:15.060951 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00127/att-02/compact.done) = No such file or directory: 7.214 sec on http probe failed: 3, retry 2024/11/19-15:20:15.465312 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-02 2024/11/19-15:20:15.465399 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00126/att-02/compact.done) = No such file or directory: 11.231 sec on http probe failed: 5, giveup 2024/11/19-15:20:15.465419 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00126/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.819 sec, real 11.524 (submit 283.909 ms wait 11.231 s), input 127.819 MiB, send shutdown 2024/11/19-15:20:15.470986 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00126/att-02 2024/11/19-15:20:15.471220 139851158521600 [dcompact_etcd.cc:1773] job-00126/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:15.473157 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00126) = success 2024/11/19-15:20:15.473166 139851158521600 [dcompact_etcd.cc:1773] job-00126/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:20:15.473188 139851158521600 (Original Log Time 2024/11/19-15:19:40.848621) EVENT_LOG_v1 {"time_micros": 1732000780848614, "job": 126, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4646, 4649, 4654, 4656], "score": 4.94465, "input_data_size": 133300236, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:15.473195 139851158521600 [compaction_job.cc:2437] [default] [JOB 126] Compacting 4@1 files to L2, score 4.94, subcompactions 1 : 1 2024/11/19-15:20:15.473203 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 124 Base level 1, inputs: [4646(31MB) 4649(31MB) 4654(31MB) 4656(31MB)] 2024/11/19-15:20:15.473464 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004779.sst) /storage/mytopling/..//tmp/Topling-82038-4Zai0W 2024/11/19-15:20:15.617828 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004779.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:20:15.643837 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-00 2024/11/19-15:20:15.643916 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00134/att-00/compact.done) = No such file or directory: 6.702 sec on http probe failed: 5, giveup 2024/11/19-15:20:15.643943 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00134/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 63.407 sec, real 7.002 (submit 294.224 ms wait 6.702 s), input 63.407 MiB, send shutdown 2024/11/19-15:20:15.645246 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-00 2024/11/19-15:20:15.645306 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00132/att-00/compact.done) = No such file or directory: 11.201 sec on http probe failed: 5, giveup 2024/11/19-15:20:15.645321 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00132/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.906 sec, real 11.514 (submit 298.434 ms wait 11.201 s), input 126.906 MiB, send shutdown 2024/11/19-15:20:15.651674 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-00 2024/11/19-15:20:15.651903 139850978166528 [dcompact_etcd.cc:1773] job-00134/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:15.652488 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004779.sst): index pass time = 0.03's, 214.105'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463336 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:20:15.654420 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-00 2024/11/19-15:20:15.654611 139851127064320 [dcompact_etcd.cc:1773] job-00132/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:15.706091 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-00 2024/11/19-15:20:15.706210 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00133/att-00/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:20:15.718863 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004779.sst): first pass time = 0.25's, 273.483'MB/sec 2024/11/19-15:20:15.718907 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004779.sst): old prealloc_size = 73819750, real_size = 55604400 2024/11/19-15:20:15.719158 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004779.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463336 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.84 avg-zval = 117.87 2024/11/19-15:20:15.719246 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004779.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.101 sec, Key+Value = 0.067 GB 2024/11/19-15:20:15.725797 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-02 2024/11/19-15:20:15.725877 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00129/att-02/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:20:15.812846 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004779.sst): second pass time = 0.00's,174068990.354'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 590.861'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3661.160'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.626'MB/sec write SST data time = 0.05's, 1119.419'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 174068990.354'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463336 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.84 avg-zval = 117.87 usrkeys = 463336 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.84 avg-zval = 117.87 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6616 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5719 value = 0.9913 dict = 0.00 all = 1.2093 } Zip/UnZip{ index = 0.0946 value = 1.0088 dict = 0.00 all = 0.8269 } ---------------------------- total value len = 6.124062 GB avg = 0.259 KB (by entry num) total key len = 0.606744 GB avg = 0.026 KB total ukey len = 0.417906 GB avg = 0.018 KB total ukey num = 0.023604811 Billion total entry num = 0.023604811 Billion write speed all = 68.094777848 MB/sec (with seq num) write speed all = 66.184320956 MB/sec (without seq num) 2024/11/19-15:20:15.816365 139851158521600 [compaction_job.cc:1948] [default] [JOB 126] Generated table #4779: 463336 keys, 55493848 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:15.816418 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000815816394, "cf_name": "default", "job": 126, "event": "table_file_creation", "file_number": 4779, "file_size": 55493848, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54611808, "index_size": 876544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12973408, "raw_average_key_size": 28, "raw_value_size": 54135456, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463336, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000815, "oldest_key_time": 0, "file_creation_time": 1732000815, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4779, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:15.830185 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004780.sst) /storage/mytopling/..//tmp/Topling-82038-cfRNoi 2024/11/19-15:20:15.931022 139851127064320 [dcompact_etcd.cc:1163] job-00132/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.276 sec, input = 126.906 MiB 2024/11/19-15:20:15.946533 139850978166528 [dcompact_etcd.cc:1163] job-00134/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.295 sec, input = 63.407 MiB 2024/11/19-15:20:15.971001 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004780.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:20:16.005022 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004780.sst): index pass time = 0.03's, 217.569'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462234 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:20:16.068176 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-00 2024/11/19-15:20:16.068283 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00136/att-00/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:20:16.074118 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004780.sst): first pass time = 0.24's, 274.353'MB/sec 2024/11/19-15:20:16.074165 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004780.sst): old prealloc_size = 73819750, real_size = 55464544 2024/11/19-15:20:16.074398 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004780.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462234 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.77 avg-zval = 117.85 2024/11/19-15:20:16.074482 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004780.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.103 sec, Key+Value = 0.067 GB 2024/11/19-15:20:16.170549 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004780.sst): second pass time = 0.00's,140930308.094'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 574.416'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3843.567'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 2.521'MB/sec write SST data time = 0.05's, 1100.598'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 140930308.094'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462234 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 117.85 usrkeys = 462234 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6596 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0554 GB } UnZip/Zip{ index = 10.5714 value = 0.9909 dict = 0.00 all = 1.2089 } Zip/UnZip{ index = 0.0946 value = 1.0092 dict = 0.00 all = 0.8272 } ---------------------------- total value len = 6.178038 GB avg = 0.257 KB (by entry num) total key len = 0.619687 GB avg = 0.026 KB total ukey len = 0.427150 GB avg = 0.018 KB total ukey num = 0.024067045 Billion total entry num = 0.024067045 Billion write speed all = 68.523811633 MB/sec (with seq num) write speed all = 66.582967402 MB/sec (without seq num) 2024/11/19-15:20:16.344329 139851158521600 [compaction_job.cc:1948] [default] [JOB 126] Generated table #4780: 462234 keys, 55354328 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:16.344390 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000816344365, "cf_name": "default", "job": 126, "event": "table_file_creation", "file_number": 4780, "file_size": 55354328, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54474320, "index_size": 874496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12942552, "raw_average_key_size": 28, "raw_value_size": 53976308, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462234, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000815, "oldest_key_time": 0, "file_creation_time": 1732000815, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4780, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:16.358301 139851158521600 [compaction_job.cc:703] job-00126: subcompact[0], size: 0.110848 G, files: /storage/mytopling/.rocksdb [004779,004780] 2024/11/19-15:20:16.359579 139851158521600 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:20:16.359677 139851158521600 (Original Log Time 2024/11/19-15:20:15.473232) EVENT_LOG_v1 {"time_micros": 1732000815473216, "job": 126, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4646, 4649, 4654, 4656], "score": 4.94465, "input_data_size": 133300236, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:16.359684 139851158521600 (Original Log Time 2024/11/19-15:20:16.359279) [compaction_job.cc:2022] [default] [JOB 126] Compacted 4@1 files to L2 => 110848176 bytes 2024/11/19-15:20:16.359687 139851158521600 (Original Log Time 2024/11/19-15:20:16.359547) [compaction_job.cc:1224] [default] compacted to: files[4 73 44 16 0 0 0] max score 3.17 (99 files need compaction), MB/sec: 150.6 rd, 125.2 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925570, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:16.359704 139851158521600 (Original Log Time 2024/11/19-15:20:16.359572) EVENT_LOG_v1 {"time_micros": 1732000816359560, "cf": "default", "job": 126, "event": "compaction_finished", "compaction_time_micros": 885086, "compaction_time_cpu_micros": 709956, "output_level": 2, "num_output_files": 2, "total_output_size": 110848176, "num_input_records": 925570, "num_output_records": 925570, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 73, 44, 16, 0, 0, 0]} 2024/11/19-15:20:16.443405 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004590.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:16.443444 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000816443440, "job": 126, "event": "table_file_deletion", "file_number": 4590} 2024/11/19-15:20:16.453007 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004593.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:16.453054 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000816453050, "job": 126, "event": "table_file_deletion", "file_number": 4593} 2024/11/19-15:20:16.463500 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004594.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:16.463528 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000816463525, "job": 126, "event": "table_file_deletion", "file_number": 4594} 2024/11/19-15:20:16.470460 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004597.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:16.470484 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000816470482, "job": 126, "event": "table_file_deletion", "file_number": 4597} 2024/11/19-15:20:16.470645 139851158521600 [compaction_job.cc:2437] [default] [JOB 137] Compacting 4@1 files to L2, score 3.17, subcompactions 1 : 1 2024/11/19-15:20:16.470654 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 135 Base level 1, inputs: [4691(31MB) 4692(31MB) 4693(31MB) 4694(31MB)] 2024/11/19-15:20:16.746330 139851158521600 [dcompact_etcd.cc:1163] job-00137/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.275 sec, input = 126.986 MiB 2024/11/19-15:20:17.069362 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-02 2024/11/19-15:20:17.069440 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00127/att-02/compact.done) = No such file or directory: 9.220 sec on http probe failed: 4, retry 2024/11/19-15:20:17.540351 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-01 2024/11/19-15:20:17.540465 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00134/att-01/compact.done) = No such file or directory: 1.585 sec on http probe failed: 1, retry 2024/11/19-15:20:17.712214 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-00 2024/11/19-15:20:17.712289 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00133/att-00/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:20:17.733703 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-02 2024/11/19-15:20:17.733779 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00129/att-02/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:20:18.077695 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-00 2024/11/19-15:20:18.077792 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00136/att-00/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:20:18.816157 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-01 2024/11/19-15:20:18.816236 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00134/att-01/compact.done) = No such file or directory: 2.862 sec on http probe failed: 2, retry 2024/11/19-15:20:19.074949 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-02 2024/11/19-15:20:19.075049 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00127/att-02/compact.done) = No such file or directory: 11.229 sec on http probe failed: 5, giveup 2024/11/19-15:20:19.075069 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00127/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.816 sec, real 11.508 (submit 274.034 ms wait 11.229 s), input 127.816 MiB, send shutdown 2024/11/19-15:20:19.083027 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00127/att-02 2024/11/19-15:20:19.083260 139851179493120 [dcompact_etcd.cc:1773] job-00127/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:19.085133 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00127) = success 2024/11/19-15:20:19.085147 139851179493120 [dcompact_etcd.cc:1773] job-00127/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:20:19.085168 139851179493120 (Original Log Time 2024/11/19-15:19:44.266966) EVENT_LOG_v1 {"time_micros": 1732000784266957, "job": 127, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4658, 4660, 4662, 4664], "score": 4.695, "input_data_size": 133300300, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:19.085176 139851179493120 [compaction_job.cc:2437] [default] [JOB 127] Compacting 4@1 files to L2, score 4.70, subcompactions 1 : 1 2024/11/19-15:20:19.085183 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 125 Base level 1, inputs: [4658(31MB) 4660(31MB) 4662(31MB) 4664(31MB)] 2024/11/19-15:20:19.085398 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004781.sst) /storage/mytopling/..//tmp/Topling-82038-Quq7mb 2024/11/19-15:20:19.112358 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-01 2024/11/19-15:20:19.112448 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00132/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:20:19.152194 139852349699840 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:20:19.250057 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004781.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:20:19.284387 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004781.sst): index pass time = 0.03's, 216.198'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463405 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:20:19.350929 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004781.sst): first pass time = 0.27's, 252.754'MB/sec 2024/11/19-15:20:19.350993 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004781.sst): old prealloc_size = 73819750, real_size = 55610288 2024/11/19-15:20:19.351224 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004781.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463405 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.82 avg-zval = 117.86 2024/11/19-15:20:19.351278 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004781.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.101 sec, Key+Value = 0.067 GB 2024/11/19-15:20:19.444725 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004781.sst): second pass time = 0.00's,128278734.597'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 592.155'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3833.748'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.623'MB/sec write SST data time = 0.05's, 1125.033'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 128278734.597'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463405 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.82 avg-zval = 117.86 usrkeys = 463405 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.82 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6616 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5711 value = 0.9911 dict = 0.00 all = 1.2092 } Zip/UnZip{ index = 0.0946 value = 1.0089 dict = 0.00 all = 0.8270 } ---------------------------- total value len = 6.232172 GB avg = 0.254 KB (by entry num) total key len = 0.632662 GB avg = 0.026 KB total ukey len = 0.436418 GB avg = 0.018 KB total ukey num = 0.024530450 Billion total entry num = 0.024530450 Billion write speed all = 66.989313196 MB/sec (with seq num) write speed all = 65.074303433 MB/sec (without seq num) 2024/11/19-15:20:19.448155 139851179493120 [compaction_job.cc:1948] [default] [JOB 127] Generated table #4781: 463405 keys, 55499736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:19.448208 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000819448184, "cf_name": "default", "job": 127, "event": "table_file_creation", "file_number": 4781, "file_size": 55499736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54617504, "index_size": 876736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12975340, "raw_average_key_size": 28, "raw_value_size": 54133626, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463405, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000819, "oldest_key_time": 0, "file_creation_time": 1732000819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4781, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:19.462228 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004782.sst) /storage/mytopling/..//tmp/Topling-82038-U6X6CR 2024/11/19-15:20:19.603816 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004782.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:20:19.637654 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004782.sst): index pass time = 0.03's, 218.722'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462090 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:20:19.705493 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004782.sst): first pass time = 0.24's, 275.094'MB/sec 2024/11/19-15:20:19.705545 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004782.sst): old prealloc_size = 73819750, real_size = 55451680 2024/11/19-15:20:19.705771 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004782.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462090 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.81 avg-zval = 117.86 2024/11/19-15:20:19.705825 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004782.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.102 sec, Key+Value = 0.067 GB 2024/11/19-15:20:19.721559 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-00 2024/11/19-15:20:19.721636 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00133/att-00/compact.done) = No such file or directory: 11.201 sec on http probe failed: 5, giveup 2024/11/19-15:20:19.721653 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00133/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.877 sec, real 11.492 (submit 281.288 ms wait 11.201 s), input 126.877 MiB, send shutdown 2024/11/19-15:20:19.730102 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-00 2024/11/19-15:20:19.730326 139851148035840 [dcompact_etcd.cc:1773] job-00133/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:19.739240 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-02 2024/11/19-15:20:19.739302 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00129/att-02/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:20:19.799712 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004782.sst): second pass time = 0.00's,167110835.913'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 587.786'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3970.678'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.605'MB/sec write SST data time = 0.05's, 1116.034'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 167110835.913'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462090 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 117.86 usrkeys = 462090 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6596 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0553 GB } UnZip/Zip{ index = 10.5712 value = 0.9911 dict = 0.00 all = 1.2091 } Zip/UnZip{ index = 0.0946 value = 1.0090 dict = 0.00 all = 0.8270 } ---------------------------- total value len = 6.286149 GB avg = 0.252 KB (by entry num) total key len = 0.645600 GB avg = 0.026 KB total ukey len = 0.445660 GB avg = 0.018 KB total ukey num = 0.024992540 Billion total entry num = 0.024992540 Billion write speed all = 67.408784116 MB/sec (with seq num) write speed all = 65.464435958 MB/sec (without seq num) 2024/11/19-15:20:19.803041 139851179493120 [compaction_job.cc:1948] [default] [JOB 127] Generated table #4782: 462090 keys, 55341464 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:19.803097 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000819803072, "cf_name": "default", "job": 127, "event": "table_file_creation", "file_number": 4782, "file_size": 55341464, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54461712, "index_size": 874240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12938520, "raw_average_key_size": 28, "raw_value_size": 53976800, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462090, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000819, "oldest_key_time": 0, "file_creation_time": 1732000819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4782, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:19.816378 139851179493120 [compaction_job.cc:703] job-00127: subcompact[0], size: 0.110841 G, files: /storage/mytopling/.rocksdb [004781,004782] 2024/11/19-15:20:19.817668 139851179493120 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:20:19.817768 139851179493120 (Original Log Time 2024/11/19-15:20:19.085214) EVENT_LOG_v1 {"time_micros": 1732000819085196, "job": 127, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4658, 4660, 4662, 4664], "score": 4.695, "input_data_size": 133300300, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:19.817774 139851179493120 (Original Log Time 2024/11/19-15:20:19.817374) [compaction_job.cc:2022] [default] [JOB 127] Compacted 4@1 files to L2 => 110841200 bytes 2024/11/19-15:20:19.817777 139851179493120 (Original Log Time 2024/11/19-15:20:19.817633) [compaction_job.cc:1224] [default] compacted to: files[4 69 46 16 0 0 0] max score 2.92 (97 files need compaction), MB/sec: 182.3 rd, 151.6 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925495, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:19.817792 139851179493120 (Original Log Time 2024/11/19-15:20:19.817661) EVENT_LOG_v1 {"time_micros": 1732000819817647, "cf": "default", "job": 127, "event": "compaction_finished", "compaction_time_micros": 731179, "compaction_time_cpu_micros": 706996, "output_level": 2, "num_output_files": 2, "total_output_size": 110841200, "num_input_records": 925495, "num_output_records": 925495, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 69, 46, 16, 0, 0, 0]} 2024/11/19-15:20:19.821605 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004636.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:19.821632 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000819821629, "job": 127, "event": "table_file_deletion", "file_number": 4636} 2024/11/19-15:20:19.825649 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004637.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:19.825671 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000819825669, "job": 127, "event": "table_file_deletion", "file_number": 4637} 2024/11/19-15:20:19.829738 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004638.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:19.829759 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000819829757, "job": 127, "event": "table_file_deletion", "file_number": 4638} 2024/11/19-15:20:19.833793 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004640.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:19.833813 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000819833811, "job": 127, "event": "table_file_deletion", "file_number": 4640} 2024/11/19-15:20:19.837781 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004642.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:19.837800 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000819837798, "job": 127, "event": "table_file_deletion", "file_number": 4642} 2024/11/19-15:20:19.841661 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004644.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:19.841680 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000819841678, "job": 127, "event": "table_file_deletion", "file_number": 4644} 2024/11/19-15:20:19.845873 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004647.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:19.845891 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000819845890, "job": 127, "event": "table_file_deletion", "file_number": 4647} 2024/11/19-15:20:19.849966 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004648.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:19.849985 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000819849983, "job": 127, "event": "table_file_deletion", "file_number": 4648} 2024/11/19-15:20:19.850155 139851179493120 [compaction_job.cc:2437] [default] [JOB 138] Compacting 4@1 files to L2, score 2.92, subcompactions 1 : 1 2024/11/19-15:20:19.850164 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 136 Base level 1, inputs: [4695(31MB) 4696(31MB) 4697(31MB) 4698(31MB)] 2024/11/19-15:20:19.929620 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-00 2024/11/19-15:20:19.929705 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00137/att-00/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:20:20.029789 139851148035840 [dcompact_etcd.cc:1163] job-00133/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.299 sec, input = 126.877 MiB 2024/11/19-15:20:20.086697 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-00 2024/11/19-15:20:20.086764 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00136/att-00/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:20:20.090143 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-01 2024/11/19-15:20:20.090184 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00134/att-01/compact.done) = No such file or directory: 4.138 sec on http probe failed: 3, retry 2024/11/19-15:20:20.158589 139851179493120 [dcompact_etcd.cc:1163] job-00138/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.308 sec, input = 126.908 MiB 2024/11/19-15:20:21.121790 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-01 2024/11/19-15:20:21.121873 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00132/att-01/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:20:21.367467 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-01 2024/11/19-15:20:21.367567 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00134/att-01/compact.done) = No such file or directory: 5.412 sec on http probe failed: 4, retry 2024/11/19-15:20:21.749335 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-02 2024/11/19-15:20:21.749440 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00129/att-02/compact.done) = No such file or directory: 11.200 sec on http probe failed: 5, giveup 2024/11/19-15:20:21.749458 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00129/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.889 sec, real 11.493 (submit 282.711 ms wait 11.200 s), input 126.889 MiB, send shutdown 2024/11/19-15:20:21.759314 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00129/att-02 2024/11/19-15:20:21.759575 139851169007360 [dcompact_etcd.cc:1773] job-00129/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:21.761496 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00129) = success 2024/11/19-15:20:21.761504 139851169007360 [dcompact_etcd.cc:1773] job-00129/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:20:21.761523 139851169007360 (Original Log Time 2024/11/19-15:19:47.239464) EVENT_LOG_v1 {"time_micros": 1732000787239456, "job": 129, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4669, 4671, 4672, 4674], "score": 4.35502, "input_data_size": 134098952, "oldest_snapshot_seqno": 125710842} 2024/11/19-15:20:21.761531 139851169007360 [compaction_job.cc:2437] [default] [JOB 129] Compacting 4@1 files to L2, score 4.36, subcompactions 1 : 1 2024/11/19-15:20:21.761539 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 127 Base level 1, inputs: [4669(31MB) 4671(31MB) 4672(31MB) 4674(31MB)] 2024/11/19-15:20:21.761769 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004783.sst) /storage/mytopling/..//tmp/Topling-82038-YjcOkP 2024/11/19-15:20:21.815595 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004783.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:20:21.815712 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004783.sst): index pass time = 0.00's,5930.240'MB/sec index type = UintIndex_AllOne usrkeys = 64713 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:20:21.884333 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004783.sst): first pass time = 0.12's, 547.605'MB/sec 2024/11/19-15:20:21.884385 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004783.sst): old prealloc_size = 73819750, real_size = 66042024 2024/11/19-15:20:21.884407 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004783.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64713 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.03 avg-zval =1020.28 2024/11/19-15:20:21.938395 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-00 2024/11/19-15:20:21.938508 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00137/att-00/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:20:21.939367 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004783.sst): second pass time = 0.00's,151298832.184'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1201.216'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.284'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.411'MB/sec write SST data time = 0.02's, 4236.387'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 151298832.184'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64713 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.03 avg-zval = 1020.28 usrkeys = 64713 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.03 avg-zval = 1020.28 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.5625 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 6.351964 GB avg = 0.253 KB (by entry num) total key len = 0.646895 GB avg = 0.026 KB total ukey len = 0.446437 GB avg = 0.018 KB total ukey num = 0.025057253 Billion total entry num = 0.025057253 Billion write speed all = 66.674084821 MB/sec (with seq num) write speed all = 64.764436939 MB/sec (without seq num) 2024/11/19-15:20:21.943148 139851169007360 [compaction_job.cc:1948] [default] [JOB 129] Generated table #4783: 64713 keys, 66026826 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:21.943201 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000821943176, "cf_name": "default", "job": 129, "event": "table_file_creation", "file_number": 4783, "file_size": 66026826, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025520, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294260, "raw_average_key_size": 20, "raw_value_size": 65814992, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64713, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000821, "oldest_key_time": 0, "file_creation_time": 1732000821, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4783, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:21.957475 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004784.sst) /storage/mytopling/..//tmp/Topling-82038-nWm98Z 2024/11/19-15:20:22.010420 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004784.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:20:22.013756 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004784.sst): index pass time = 0.00's, 154.035'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63593 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:20:22.076287 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004784.sst): first pass time = 0.12's, 555.108'MB/sec 2024/11/19-15:20:22.076343 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004784.sst): old prealloc_size = 73819750, real_size = 65002624 2024/11/19-15:20:22.076401 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004784.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63593 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1016.96 avg-zval =1020.21 2024/11/19-15:20:22.076471 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004784.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.066 GB 2024/11/19-15:20:22.096442 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-00 2024/11/19-15:20:22.096522 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00136/att-00/compact.done) = No such file or directory: 9.200 sec on http probe failed: 4, retry 2024/11/19-15:20:22.135484 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004784.sst): second pass time = 0.00's,171088126.984'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1099.088'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1997.566'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.377'MB/sec write SST data time = 0.02's, 3840.854'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 171088126.984'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63593 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.96 avg-zval = 1020.21 usrkeys = 63593 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.96 avg-zval = 1020.21 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7924 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0429 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 6.416635 GB avg = 0.255 KB (by entry num) total key len = 0.648167 GB avg = 0.026 KB total ukey len = 0.447200 GB avg = 0.018 KB total ukey num = 0.025120846 Billion total entry num = 0.025120846 Billion write speed all = 67.176781280 MB/sec (with seq num) write speed all = 65.265857059 MB/sec (without seq num) 2024/11/19-15:20:22.139547 139851169007360 [compaction_job.cc:1948] [default] [JOB 129] Generated table #4784: 63593 keys, 64988100 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:22.139599 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000822139576, "cf_name": "default", "job": 129, "event": "table_file_creation", "file_number": 4784, "file_size": 64988100, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64878192, "index_size": 108352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1271860, "raw_average_key_size": 20, "raw_value_size": 64671312, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63593, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000821, "oldest_key_time": 0, "file_creation_time": 1732000821, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4784, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:22.153712 139851169007360 [compaction_job.cc:703] job-00129: subcompact[0], size: 0.131015 G, files: /storage/mytopling/.rocksdb [004783,004784] 2024/11/19-15:20:22.154448 139851169007360 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:20:22.154541 139851169007360 (Original Log Time 2024/11/19-15:20:21.761571) EVENT_LOG_v1 {"time_micros": 1732000821761550, "job": 129, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4669, 4671, 4672, 4674], "score": 4.35502, "input_data_size": 134098952, "oldest_snapshot_seqno": 125710842} 2024/11/19-15:20:22.154547 139851169007360 (Original Log Time 2024/11/19-15:20:22.154154) [compaction_job.cc:2022] [default] [JOB 129] Compacted 4@1 files to L2 => 131014926 bytes 2024/11/19-15:20:22.154550 139851169007360 (Original Log Time 2024/11/19-15:20:22.154414) [compaction_job.cc:1224] [default] compacted to: files[4 65 48 16 0 0 0] max score 2.67 (95 files need compaction), MB/sec: 342.0 rd, 334.1 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(124.9 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128306, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:22.154565 139851169007360 (Original Log Time 2024/11/19-15:20:22.154441) EVENT_LOG_v1 {"time_micros": 1732000822154428, "cf": "default", "job": 129, "event": "compaction_finished", "compaction_time_micros": 392157, "compaction_time_cpu_micros": 389863, "output_level": 2, "num_output_files": 2, "total_output_size": 131014926, "num_input_records": 128306, "num_output_records": 128306, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 65, 48, 16, 0, 0, 0]} 2024/11/19-15:20:22.158488 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004652.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:22.158516 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000822158513, "job": 129, "event": "table_file_deletion", "file_number": 4652} 2024/11/19-15:20:22.162705 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004655.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:22.162730 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000822162727, "job": 129, "event": "table_file_deletion", "file_number": 4655} 2024/11/19-15:20:22.166744 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004657.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:22.166766 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000822166764, "job": 129, "event": "table_file_deletion", "file_number": 4657} 2024/11/19-15:20:22.171150 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004659.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:22.171173 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000822171171, "job": 129, "event": "table_file_deletion", "file_number": 4659} 2024/11/19-15:20:22.175428 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004666.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:22.175449 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000822175447, "job": 129, "event": "table_file_deletion", "file_number": 4666} 2024/11/19-15:20:22.177352 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004668.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:22.177371 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000822177369, "job": 129, "event": "table_file_deletion", "file_number": 4668} 2024/11/19-15:20:22.177536 139851169007360 [compaction_job.cc:2437] [default] [JOB 139] Compacting 4@1 files to L2, score 2.67, subcompactions 1 : 1 2024/11/19-15:20:22.177544 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 137 Base level 1, inputs: [4699(31MB) 4700(31MB) 4701(31MB) 4702(31MB)] 2024/11/19-15:20:22.495715 139851169007360 [dcompact_etcd.cc:1163] job-00139/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.318 sec, input = 126.850 MiB 2024/11/19-15:20:22.644561 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-01 2024/11/19-15:20:22.644637 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00134/att-01/compact.done) = No such file or directory: 6.689 sec on http probe failed: 5, giveup 2024/11/19-15:20:22.644654 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00134/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 63.407 sec, real 6.993 (submit 294.530 ms wait 6.689 s), input 63.407 MiB, send shutdown 2024/11/19-15:20:22.653963 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-01 2024/11/19-15:20:22.654182 139850978166528 [dcompact_etcd.cc:1773] job-00134/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:22.946435 139850978166528 [dcompact_etcd.cc:1163] job-00134/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.292 sec, input = 63.407 MiB 2024/11/19-15:20:23.130913 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-01 2024/11/19-15:20:23.131020 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00132/att-01/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:20:23.210115 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-01 2024/11/19-15:20:23.210226 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00133/att-01/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:20:23.339221 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-00 2024/11/19-15:20:23.339320 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00138/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:20:23.948080 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-00 2024/11/19-15:20:23.948179 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00137/att-00/compact.done) = No such file or directory: 7.192 sec on http probe failed: 3, retry 2024/11/19-15:20:24.105963 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-00 2024/11/19-15:20:24.106047 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00136/att-00/compact.done) = No such file or directory: 11.209 sec on http probe failed: 5, giveup 2024/11/19-15:20:24.106067 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00136/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.987 sec, real 11.507 (submit 288.416 ms wait 11.209 s), input 126.987 MiB, send shutdown 2024/11/19-15:20:24.114910 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-00 2024/11/19-15:20:24.115140 139851137550080 [dcompact_etcd.cc:1773] job-00136/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:24.387701 139851137550080 [dcompact_etcd.cc:1163] job-00136/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.272 sec, input = 126.987 MiB 2024/11/19-15:20:24.540929 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-02 2024/11/19-15:20:24.541024 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00134/att-02/compact.done) = No such file or directory: 1.585 sec on http probe failed: 1, retry 2024/11/19-15:20:25.140039 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-01 2024/11/19-15:20:25.140144 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00132/att-01/compact.done) = No such file or directory: 9.200 sec on http probe failed: 4, retry 2024/11/19-15:20:25.217226 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-01 2024/11/19-15:20:25.217328 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00133/att-01/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:20:25.348784 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-00 2024/11/19-15:20:25.348879 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00138/att-00/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:20:25.676376 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-00 2024/11/19-15:20:25.676516 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00139/att-00/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:20:25.817555 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-02 2024/11/19-15:20:25.817627 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00134/att-02/compact.done) = No such file or directory: 2.863 sec on http probe failed: 2, retry 2024/11/19-15:20:25.956816 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-00 2024/11/19-15:20:25.956916 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00137/att-00/compact.done) = No such file or directory: 9.202 sec on http probe failed: 4, retry 2024/11/19-15:20:26.830171 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4785. Immutable memtables: 0. 2024/11/19-15:20:26.830257 139852362282752 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:20:26.830534 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 140] Syncing log #4776 2024/11/19-15:20:26.830654 139850957195008 (Original Log Time 2024/11/19-15:20:26.830319) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:20:26.830666 139850957195008 [flush_job.cc:879] [default] [JOB 140] Flushing memtable with next log file: 4785 2024/11/19-15:20:26.830693 139850957195008 EVENT_LOG_v1 {"time_micros": 1732000826830686, "job": 140, "event": "flush_started", "num_memtables": 1, "num_entries": 1188111, "num_deletes": 0, "total_data_size": 472253956, "memory_usage": 537394568, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:20:26.830699 139850957195008 [flush_job.cc:910] [default] [JOB 140] Level-0 flush table #4786: started 2024/11/19-15:20:26.831512 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000016.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004786.sst in ConvertToSST, mem_size = 512.499 M, time = 800.534 us 2024/11/19-15:20:27.094807 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-02 2024/11/19-15:20:27.094961 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00134/att-02/compact.done) = No such file or directory: 4.139 sec on http probe failed: 3, retry 2024/11/19-15:20:27.148431 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-01 2024/11/19-15:20:27.148504 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00132/att-01/compact.done) = No such file or directory: 11.209 sec on http probe failed: 5, giveup 2024/11/19-15:20:27.148522 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00132/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.906 sec, real 11.494 (submit 276.329 ms wait 11.209 s), input 126.906 MiB, send shutdown 2024/11/19-15:20:27.153900 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-01 2024/11/19-15:20:27.192175 139851127064320 [dcompact_etcd.cc:1773] job-00132/att-01: CleanFiles 0.038 sec : 0.000 2024/11/19-15:20:27.226354 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-01 2024/11/19-15:20:27.226486 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00133/att-01/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:20:27.348705 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004786.sst): fsize = 512.501 M, time(ms): open: 0.044, seek: 0.019, finish: 0.049, meta: 0.016, Flush: 0.017, sync: 517.009, close: 0.013, all: 517.167 2024/11/19-15:20:27.350091 139850957195008 [flush_job.cc:1083] [default] [JOB 140] Flush lasted 519437 microseconds, and 438722 cpu microseconds. 2024/11/19-15:20:27.354267 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-00 2024/11/19-15:20:27.354361 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00138/att-00/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:20:27.370459 139850957195008 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:20:27.370548 139850957195008 (Original Log Time 2024/11/19-15:20:27.348758) [flush_job.cc:1029] [default] [JOB 140] Level-0 flush table #4786: 537395994 bytes OK (needs compaction) 2024/11/19-15:20:27.370554 139850957195008 (Original Log Time 2024/11/19-15:20:27.350112) [memtable_list.cc:560] [default] Level-0 commit table #4786 started 2024/11/19-15:20:27.370557 139850957195008 (Original Log Time 2024/11/19-15:20:27.370426) [memtable_list.cc:759] [default] Level-0 commit table #4786: memtable #1 done 2024/11/19-15:20:27.370559 139850957195008 (Original Log Time 2024/11/19-15:20:27.370452) EVENT_LOG_v1 {"time_micros": 1732000827370446, "job": 140, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [5, 65, 48, 16, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:20:27.370562 139850957195008 (Original Log Time 2024/11/19-15:20:27.370518) [db_impl_compaction_flush.cc:310] [default] Level summary: files[5 65 48 16 0 0 0] max score 2.42 (92 files need compaction) 2024/11/19-15:20:27.499037 139851127064320 [dcompact_etcd.cc:1163] job-00132/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.307 sec, input = 126.906 MiB 2024/11/19-15:20:27.570557 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-01 2024/11/19-15:20:27.570625 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00136/att-01/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:20:27.683225 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-00 2024/11/19-15:20:27.683313 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00139/att-00/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:20:27.964918 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-00 2024/11/19-15:20:27.965007 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00137/att-00/compact.done) = No such file or directory: 11.211 sec on http probe failed: 5, giveup 2024/11/19-15:20:27.965026 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00137/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.986 sec, real 11.494 (submit 275.188 ms wait 11.211 s), input 126.986 MiB, send shutdown 2024/11/19-15:20:27.973070 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-00 2024/11/19-15:20:27.973295 139851158521600 [dcompact_etcd.cc:1773] job-00137/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:28.237964 139851158521600 [dcompact_etcd.cc:1163] job-00137/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.265 sec, input = 126.986 MiB 2024/11/19-15:20:28.370909 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-02 2024/11/19-15:20:28.370990 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00134/att-02/compact.done) = No such file or directory: 5.417 sec on http probe failed: 4, retry 2024/11/19-15:20:29.235827 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-01 2024/11/19-15:20:29.235914 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00133/att-01/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:20:29.363326 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-00 2024/11/19-15:20:29.363425 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00138/att-00/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:20:29.577250 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-01 2024/11/19-15:20:29.577328 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00136/att-01/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:20:29.647894 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-02 2024/11/19-15:20:29.647980 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00134/att-02/compact.done) = No such file or directory: 6.693 sec on http probe failed: 5, giveup 2024/11/19-15:20:29.647999 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00134/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 63.407 sec, real 6.994 (submit 292.166 ms wait 6.693 s), input 63.407 MiB, send shutdown 2024/11/19-15:20:29.657522 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00134/att-02 2024/11/19-15:20:29.657761 139850978166528 [dcompact_etcd.cc:1773] job-00134/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:29.659649 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00134) = success 2024/11/19-15:20:29.659664 139850978166528 [dcompact_etcd.cc:1773] job-00134/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:20:29.659686 139850978166528 (Original Log Time 2024/11/19-15:20:08.641792) EVENT_LOG_v1 {"time_micros": 1732000808641785, "job": 134, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4684, 4685], "score": 3.54324, "input_data_size": 67049508, "oldest_snapshot_seqno": 127426644} 2024/11/19-15:20:29.659693 139850978166528 [compaction_job.cc:2437] [default] [JOB 134] Compacting 2@1 files to L2, score 3.54, subcompactions 1 : 1 2024/11/19-15:20:29.659701 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 132 Base level 1, inputs: [4684(31MB) 4685(31MB)] 2024/11/19-15:20:29.659953 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004787.sst) /storage/mytopling/..//tmp/Topling-82038-fpPMH0 2024/11/19-15:20:29.690876 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-00 2024/11/19-15:20:29.691041 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00139/att-00/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:20:29.713809 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004787.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:20:29.718914 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004787.sst): index pass time = 0.01's, 101.224'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64114 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:20:29.779581 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004787.sst): first pass time = 0.12's, 555.853'MB/sec 2024/11/19-15:20:29.779636 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004787.sst): old prealloc_size = 73754458, real_size = 65538352 2024/11/19-15:20:29.779690 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004787.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64114 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0652 GB zip-val = 0.0654 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:20:29.779738 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004787.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.066 GB 2024/11/19-15:20:29.838642 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004787.sst): second pass time = 0.00's,89813185.950'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1110.684'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2162.573'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.380'MB/sec write SST data time = 0.02's, 3917.091'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 89813185.950'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64114 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 64114 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7989 UnZipSize{ index = 0.0008 GB value = 0.0652 GB dict = 0.00 MB all = 0.0665 GB } __ZipSize{ index = 0.0001 GB value = 0.0654 GB dict = 0.00 MB all = 0.0655 GB } UnZip/Zip{ index = 7.0465 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 6.481840 GB avg = 0.257 KB (by entry num) total key len = 0.649449 GB avg = 0.026 KB total ukey len = 0.447969 GB avg = 0.018 KB total ukey num = 0.025184960 Billion total entry num = 0.025184960 Billion write speed all = 63.181167321 MB/sec (with seq num) write speed all = 61.396115195 MB/sec (without seq num) 2024/11/19-15:20:29.842319 139850978166528 [compaction_job.cc:1948] [default] [JOB 134] Generated table #4787: 64114 keys, 65523716 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:29.842373 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000829842347, "cf_name": "default", "job": 134, "event": "table_file_creation", "file_number": 4787, "file_size": 65523716, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 65412960, "index_size": 109184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1282280, "raw_average_key_size": 20, "raw_value_size": 65204373, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64114, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000829, "oldest_key_time": 0, "file_creation_time": 1732000829, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4787, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:29.856354 139850978166528 [compaction_job.cc:703] job-00134: subcompact[0], size: 0.065524 G, files: /storage/mytopling/.rocksdb [004787] 2024/11/19-15:20:29.856953 139850978166528 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:20:29.857096 139850978166528 (Original Log Time 2024/11/19-15:20:29.659734) EVENT_LOG_v1 {"time_micros": 1732000829659713, "job": 134, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4684, 4685], "score": 3.54324, "input_data_size": 67049508, "oldest_snapshot_seqno": 127426644} 2024/11/19-15:20:29.857102 139850978166528 (Original Log Time 2024/11/19-15:20:29.856646) [compaction_job.cc:2022] [default] [JOB 134] Compacted 2@1 files to L2 => 65523716 bytes 2024/11/19-15:20:29.857105 139850978166528 (Original Log Time 2024/11/19-15:20:29.856908) [compaction_job.cc:1224] [default] compacted to: files[5 63 49 16 0 0 0] max score 2.42 (93 files need compaction), MB/sec: 341.0 rd, 333.2 wr, level 2, files in(2, 0) out(1 +0 blob) MB in(63.9, 0.0 +0.0 blob) out(62.5 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 64114, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:29.857124 139850978166528 (Original Log Time 2024/11/19-15:20:29.856946) EVENT_LOG_v1 {"time_micros": 1732000829856930, "cf": "default", "job": 134, "event": "compaction_finished", "compaction_time_micros": 196632, "compaction_time_cpu_micros": 195823, "output_level": 2, "num_output_files": 1, "total_output_size": 65523716, "num_input_records": 64114, "num_output_records": 64114, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 63, 49, 16, 0, 0, 0]} 2024/11/19-15:20:29.857195 139850978166528 [compaction_job.cc:2437] [default] [JOB 141] Compacting 4@1 files to L2, score 2.42, subcompactions 1 : 1 2024/11/19-15:20:29.857203 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 139 Base level 1, inputs: [4703(31MB) 4704(31MB) 4705(31MB) 4706(31MB)] 2024/11/19-15:20:30.150775 139850978166528 [dcompact_etcd.cc:1163] job-00141/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.293 sec, input = 126.972 MiB 2024/11/19-15:20:30.681049 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-02 2024/11/19-15:20:30.681163 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00132/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:20:31.244574 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-01 2024/11/19-15:20:31.244648 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00133/att-01/compact.done) = No such file or directory: 11.206 sec on http probe failed: 5, giveup 2024/11/19-15:20:31.244667 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00133/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.877 sec, real 11.514 (submit 299.387 ms wait 11.206 s), input 126.877 MiB, send shutdown 2024/11/19-15:20:31.252367 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-01 2024/11/19-15:20:31.252603 139851148035840 [dcompact_etcd.cc:1773] job-00133/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:31.371487 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-00 2024/11/19-15:20:31.371560 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00138/att-00/compact.done) = No such file or directory: 11.205 sec on http probe failed: 5, giveup 2024/11/19-15:20:31.371579 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00138/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.908 sec, real 11.521 (submit 307.967 ms wait 11.205 s), input 126.908 MiB, send shutdown 2024/11/19-15:20:31.379940 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-00 2024/11/19-15:20:31.380205 139851179493120 [dcompact_etcd.cc:1773] job-00138/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:31.421458 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-01 2024/11/19-15:20:31.421522 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00137/att-01/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:20:31.540833 139851148035840 [dcompact_etcd.cc:1163] job-00133/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.288 sec, input = 126.877 MiB 2024/11/19-15:20:31.585103 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-01 2024/11/19-15:20:31.585163 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00136/att-01/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:20:31.647231 139851179493120 [dcompact_etcd.cc:1163] job-00138/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.267 sec, input = 126.908 MiB 2024/11/19-15:20:31.700231 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-00 2024/11/19-15:20:31.700328 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00139/att-00/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:20:32.686903 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-02 2024/11/19-15:20:32.687015 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00132/att-02/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:20:33.334411 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-00 2024/11/19-15:20:33.334526 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00141/att-00/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:20:33.426311 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-01 2024/11/19-15:20:33.426393 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00137/att-01/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:20:33.591961 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-01 2024/11/19-15:20:33.592035 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00136/att-01/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:20:33.709518 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-00 2024/11/19-15:20:33.709598 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00139/att-00/compact.done) = No such file or directory: 11.205 sec on http probe failed: 5, giveup 2024/11/19-15:20:33.709617 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00139/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.850 sec, real 11.532 (submit 317.685 ms wait 11.205 s), input 126.850 MiB, send shutdown 2024/11/19-15:20:33.717694 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-00 2024/11/19-15:20:33.717930 139851169007360 [dcompact_etcd.cc:1773] job-00139/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:33.986449 139851169007360 [dcompact_etcd.cc:1163] job-00139/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.268 sec, input = 126.850 MiB 2024/11/19-15:20:34.696007 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-02 2024/11/19-15:20:34.696108 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00132/att-02/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:20:34.723403 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-02 2024/11/19-15:20:34.723505 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00133/att-02/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:20:34.825187 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-01 2024/11/19-15:20:34.825264 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00138/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:20:35.339449 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-00 2024/11/19-15:20:35.339542 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00141/att-00/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:20:35.434825 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-01 2024/11/19-15:20:35.434900 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00137/att-01/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:20:35.600245 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-01 2024/11/19-15:20:35.600331 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00136/att-01/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:20:35.600349 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00136/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.987 sec, real 11.485 (submit 272.475 ms wait 11.204 s), input 126.987 MiB, send shutdown 2024/11/19-15:20:35.605864 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-01 2024/11/19-15:20:35.606097 139851137550080 [dcompact_etcd.cc:1773] job-00136/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:35.876150 139851137550080 [dcompact_etcd.cc:1163] job-00136/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.270 sec, input = 126.987 MiB 2024/11/19-15:20:36.702071 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-02 2024/11/19-15:20:36.702172 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00132/att-02/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:20:36.733383 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-02 2024/11/19-15:20:36.733459 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00133/att-02/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:20:36.832236 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-01 2024/11/19-15:20:36.832308 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00138/att-01/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:20:37.166724 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-01 2024/11/19-15:20:37.166845 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00139/att-01/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:20:37.348530 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-00 2024/11/19-15:20:37.348636 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00141/att-00/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:20:37.444527 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-01 2024/11/19-15:20:37.444608 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00137/att-01/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:20:38.707676 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-02 2024/11/19-15:20:38.707769 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00132/att-02/compact.done) = No such file or directory: 11.203 sec on http probe failed: 5, giveup 2024/11/19-15:20:38.707787 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00132/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.906 sec, real 11.515 (submit 306.638 ms wait 11.203 s), input 126.906 MiB, send shutdown 2024/11/19-15:20:38.715180 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00132/att-02 2024/11/19-15:20:38.715417 139851127064320 [dcompact_etcd.cc:1773] job-00132/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:38.717321 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00132) = success 2024/11/19-15:20:38.717333 139851127064320 [dcompact_etcd.cc:1773] job-00132/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:20:38.717355 139851127064320 (Original Log Time 2024/11/19-15:20:04.131163) EVENT_LOG_v1 {"time_micros": 1732000804131157, "job": 132, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4676, 4677, 4678, 4679], "score": 4.04279, "input_data_size": 134098312, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:38.717367 139851127064320 [compaction_job.cc:2437] [default] [JOB 132] Compacting 4@1 files to L2, score 4.04, subcompactions 1 : 1 2024/11/19-15:20:38.717376 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 130 Base level 1, inputs: [4676(31MB) 4677(31MB) 4678(31MB) 4679(31MB)] 2024/11/19-15:20:38.717618 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004788.sst) /storage/mytopling/..//tmp/Topling-82038-02GiZL 2024/11/19-15:20:38.742263 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-02 2024/11/19-15:20:38.742338 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00133/att-02/compact.done) = No such file or directory: 7.193 sec on http probe failed: 3, retry 2024/11/19-15:20:38.770882 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004788.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:20:38.774968 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004788.sst): index pass time = 0.00's, 127.438'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64714 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:20:38.837296 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004788.sst): first pass time = 0.12's, 560.812'MB/sec 2024/11/19-15:20:38.837351 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004788.sst): old prealloc_size = 73819750, real_size = 66151872 2024/11/19-15:20:38.837408 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004788.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64714 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:20:38.837462 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004788.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.067 GB 2024/11/19-15:20:38.841500 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-01 2024/11/19-15:20:38.841636 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00138/att-01/compact.done) = No such file or directory: 7.185 sec on http probe failed: 3, retry 2024/11/19-15:20:38.897127 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004788.sst): second pass time = 0.00's,217929794.702'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1106.686'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2106.301'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.376'MB/sec write SST data time = 0.02's, 3968.374'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 217929794.702'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64714 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 64714 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0464 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 6.547654 GB avg = 0.259 KB (by entry num) total key len = 0.650743 GB avg = 0.026 KB total ukey len = 0.448746 GB avg = 0.018 KB total ukey num = 0.025249674 Billion total entry num = 0.025249674 Billion write speed all = 59.037636720 MB/sec (with seq num) write speed all = 57.380955657 MB/sec (without seq num) 2024/11/19-15:20:38.900684 139851127064320 [compaction_job.cc:1948] [default] [JOB 132] Generated table #4788: 64714 keys, 66137092 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:38.900736 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000838900712, "cf_name": "default", "job": 132, "event": "table_file_creation", "file_number": 4788, "file_size": 66137092, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025328, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294280, "raw_average_key_size": 20, "raw_value_size": 65814798, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64714, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000838, "oldest_key_time": 0, "file_creation_time": 1732000838, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4788, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:38.914955 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004789.sst) /storage/mytopling/..//tmp/Topling-82038-pFlTWT 2024/11/19-15:20:38.966715 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004789.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:20:38.966819 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004789.sst): index pass time = 0.00's,7514.332'MB/sec index type = UintIndex_AllOne usrkeys = 63606 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:20:39.032158 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004789.sst): first pass time = 0.12's, 562.882'MB/sec 2024/11/19-15:20:39.032216 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004789.sst): old prealloc_size = 73819750, real_size = 64912136 2024/11/19-15:20:39.032240 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004789.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63606 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1017.03 avg-zval =1020.28 2024/11/19-15:20:39.061301 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-02 2024/11/19-15:20:39.061417 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00136/att-02/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:20:39.085867 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004789.sst): second pass time = 0.00's,157777929.268'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1209.901'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.481'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.412'MB/sec write SST data time = 0.01's, 4330.016'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 157777929.268'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63606 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.03 avg-zval = 1020.28 usrkeys = 63606 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.03 avg-zval = 1020.28 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0000 GB value = 0.0649 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index =3975.3750 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 6.612343 GB avg = 0.261 KB (by entry num) total key len = 0.652015 GB avg = 0.026 KB total ukey len = 0.449509 GB avg = 0.018 KB total ukey num = 0.025313280 Billion total entry num = 0.025313280 Billion write speed all = 59.486532436 MB/sec (with seq num) write speed all = 57.828245050 MB/sec (without seq num) 2024/11/19-15:20:39.089390 139851127064320 [compaction_job.cc:1948] [default] [JOB 132] Generated table #4789: 63606 keys, 64897226 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:39.089441 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000839089418, "cf_name": "default", "job": 132, "event": "table_file_creation", "file_number": 4789, "file_size": 64897226, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64895888, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1272120, "raw_average_key_size": 20, "raw_value_size": 64688951, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63606, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000838, "oldest_key_time": 0, "file_creation_time": 1732000838, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4789, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:39.103346 139851127064320 [compaction_job.cc:703] job-00132: subcompact[0], size: 0.131034 G, files: /storage/mytopling/.rocksdb [004788,004789] 2024/11/19-15:20:39.104106 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:20:39.104258 139851127064320 (Original Log Time 2024/11/19-15:20:38.717406) EVENT_LOG_v1 {"time_micros": 1732000838717389, "job": 132, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4676, 4677, 4678, 4679], "score": 4.04279, "input_data_size": 134098312, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:39.104264 139851127064320 (Original Log Time 2024/11/19-15:20:39.103749) [compaction_job.cc:2022] [default] [JOB 132] Compacted 4@1 files to L2 => 131034318 bytes 2024/11/19-15:20:39.104267 139851127064320 (Original Log Time 2024/11/19-15:20:39.104076) [compaction_job.cc:1224] [default] compacted to: files[5 59 51 16 0 0 0] max score 2.17 (91 files need compaction), MB/sec: 347.4 rd, 339.5 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128320, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:39.104283 139851127064320 (Original Log Time 2024/11/19-15:20:39.104099) EVENT_LOG_v1 {"time_micros": 1732000839104088, "cf": "default", "job": 132, "event": "compaction_finished", "compaction_time_micros": 385951, "compaction_time_cpu_micros": 384221, "output_level": 2, "num_output_files": 2, "total_output_size": 131034318, "num_input_records": 128320, "num_output_records": 128320, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 59, 51, 16, 0, 0, 0]} 2024/11/19-15:20:39.108007 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004633.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:39.108030 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000839108027, "job": 132, "event": "table_file_deletion", "file_number": 4633} 2024/11/19-15:20:39.111688 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004639.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:39.111709 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000839111707, "job": 132, "event": "table_file_deletion", "file_number": 4639} 2024/11/19-15:20:39.115750 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004670.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:39.115769 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000839115767, "job": 132, "event": "table_file_deletion", "file_number": 4670} 2024/11/19-15:20:39.117102 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004673.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:39.117119 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000839117117, "job": 132, "event": "table_file_deletion", "file_number": 4673} 2024/11/19-15:20:39.117234 139851127064320 [compaction_job.cc:2437] [default] [JOB 142] Compacting 4@1 files to L2, score 2.17, subcompactions 1 : 1 2024/11/19-15:20:39.117243 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 140 Base level 1, inputs: [4707(31MB) 4708(31MB) 4709(31MB) 4710(31MB)] 2024/11/19-15:20:39.174383 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-01 2024/11/19-15:20:39.174472 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00139/att-01/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:20:39.357454 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-00 2024/11/19-15:20:39.357551 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00141/att-00/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:20:39.441753 139851127064320 [dcompact_etcd.cc:1163] job-00142/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.324 sec, input = 126.813 MiB 2024/11/19-15:20:39.453961 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-01 2024/11/19-15:20:39.454023 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00137/att-01/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:20:39.454039 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00137/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.986 sec, real 11.481 (submit 264.591 ms wait 11.207 s), input 126.986 MiB, send shutdown 2024/11/19-15:20:39.459569 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-01 2024/11/19-15:20:39.459799 139851158521600 [dcompact_etcd.cc:1773] job-00137/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:39.670536 139852362282752 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16273899 2024/11/19-15:20:39.730227 139851158521600 [dcompact_etcd.cc:1163] job-00137/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.270 sec, input = 126.986 MiB 2024/11/19-15:20:40.750824 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-02 2024/11/19-15:20:40.750931 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00133/att-02/compact.done) = No such file or directory: 9.202 sec on http probe failed: 4, retry 2024/11/19-15:20:40.850734 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-01 2024/11/19-15:20:40.850805 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00138/att-01/compact.done) = No such file or directory: 9.194 sec on http probe failed: 4, retry 2024/11/19-15:20:41.071139 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-02 2024/11/19-15:20:41.071244 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00136/att-02/compact.done) = No such file or directory: 5.185 sec on http probe failed: 2, retry 2024/11/19-15:20:41.179439 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-01 2024/11/19-15:20:41.179518 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00139/att-01/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:20:41.366350 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-00 2024/11/19-15:20:41.366434 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00141/att-00/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:20:41.366452 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00141/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.972 sec, real 11.509 (submit 293.121 ms wait 11.207 s), input 126.972 MiB, send shutdown 2024/11/19-15:20:41.371098 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-00 2024/11/19-15:20:41.371310 139850978166528 [dcompact_etcd.cc:1773] job-00141/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:41.644932 139850978166528 [dcompact_etcd.cc:1163] job-00141/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.274 sec, input = 126.972 MiB 2024/11/19-15:20:42.617556 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-00 2024/11/19-15:20:42.617637 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00142/att-00/compact.done) = No such file or directory: 3.170 sec on http probe failed: 1, retry 2024/11/19-15:20:42.756229 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-02 2024/11/19-15:20:42.756293 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00133/att-02/compact.done) = No such file or directory: 11.210 sec on http probe failed: 5, giveup 2024/11/19-15:20:42.756311 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00133/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.877 sec, real 11.504 (submit 288.158 ms wait 11.210 s), input 126.877 MiB, send shutdown 2024/11/19-15:20:42.763234 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00133/att-02 2024/11/19-15:20:42.763452 139851148035840 [dcompact_etcd.cc:1773] job-00133/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:42.765611 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00133) = success 2024/11/19-15:20:42.765618 139851148035840 [dcompact_etcd.cc:1773] job-00133/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:20:42.765638 139851148035840 (Original Log Time 2024/11/19-15:20:08.229197) EVENT_LOG_v1 {"time_micros": 1732000808229191, "job": 133, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4680, 4681, 4682, 4683], "score": 3.79302, "input_data_size": 134100552, "oldest_snapshot_seqno": 127414640} 2024/11/19-15:20:42.765646 139851148035840 [compaction_job.cc:2437] [default] [JOB 133] Compacting 4@1 files to L2, score 3.79, subcompactions 1 : 1 2024/11/19-15:20:42.765654 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 131 Base level 1, inputs: [4680(31MB) 4681(31MB) 4682(31MB) 4683(31MB)] 2024/11/19-15:20:42.765912 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004790.sst) /storage/mytopling/..//tmp/Topling-82038-CbcwIy 2024/11/19-15:20:42.819582 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004790.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:20:42.824975 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004790.sst): index pass time = 0.01's, 96.702'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64715 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:20:42.856374 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-01 2024/11/19-15:20:42.856462 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00138/att-01/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:20:42.856479 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00138/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.908 sec, real 11.476 (submit 266.908 ms wait 11.204 s), input 126.908 MiB, send shutdown 2024/11/19-15:20:42.862235 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-01 2024/11/19-15:20:42.894795 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004790.sst): first pass time = 0.13's, 520.803'MB/sec 2024/11/19-15:20:42.894846 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004790.sst): old prealloc_size = 73819750, real_size = 66152640 2024/11/19-15:20:42.894904 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004790.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64715 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:20:42.894960 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004790.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.075 sec, Key+Value = 0.067 GB 2024/11/19-15:20:42.912930 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-02 2024/11/19-15:20:42.913013 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00137/att-02/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:20:42.941492 139851179493120 [dcompact_etcd.cc:1773] job-00138/att-01: CleanFiles 0.079 sec : 0.000 2024/11/19-15:20:42.955300 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004790.sst): second pass time = 0.00's,151998990.762'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1094.212'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2039.907'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.372'MB/sec write SST data time = 0.02's, 3907.717'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 151998990.762'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64715 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 64715 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0465 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 6.678159 GB avg = 0.263 KB (by entry num) total key len = 0.653310 GB avg = 0.026 KB total ukey len = 0.450286 GB avg = 0.018 KB total ukey num = 0.025377995 Billion total entry num = 0.025377995 Billion write speed all = 58.192198744 MB/sec (with seq num) write speed all = 56.580732923 MB/sec (without seq num) 2024/11/19-15:20:43.020910 139851148035840 [compaction_job.cc:1948] [default] [JOB 133] Generated table #4790: 64715 keys, 66137860 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:43.020970 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000843020946, "cf_name": "default", "job": 133, "event": "table_file_creation", "file_number": 4790, "file_size": 66137860, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66026096, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294300, "raw_average_key_size": 20, "raw_value_size": 65815563, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64715, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000842, "oldest_key_time": 0, "file_creation_time": 1732000842, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4790, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:43.041672 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004791.sst) /storage/mytopling/..//tmp/Topling-82038-ZD3IrF 2024/11/19-15:20:43.081548 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-02 2024/11/19-15:20:43.081634 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00136/att-02/compact.done) = No such file or directory: 7.195 sec on http probe failed: 3, retry 2024/11/19-15:20:43.094477 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004791.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:20:43.097729 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004791.sst): index pass time = 0.00's, 157.724'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63575 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:20:43.161576 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004791.sst): first pass time = 0.12's, 549.954'MB/sec 2024/11/19-15:20:43.161631 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004791.sst): old prealloc_size = 73819750, real_size = 64989696 2024/11/19-15:20:43.161688 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004791.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63575 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1017.04 avg-zval =1020.30 2024/11/19-15:20:43.161737 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004791.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.066 GB 2024/11/19-15:20:43.188491 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-01 2024/11/19-15:20:43.188653 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00139/att-01/compact.done) = No such file or directory: 9.193 sec on http probe failed: 4, retry 2024/11/19-15:20:43.226878 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004791.sst): second pass time = 0.00's,86557539.491'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 996.020'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2026.006'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.327'MB/sec write SST data time = 0.02's, 3922.867'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 86557539.491'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63575 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.04 avg-zval = 1020.30 usrkeys = 63575 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.04 avg-zval = 1020.30 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7923 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0451 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 6.742817 GB avg = 0.265 KB (by entry num) total key len = 0.654581 GB avg = 0.026 KB total ukey len = 0.451048 GB avg = 0.018 KB total ukey num = 0.025441570 Billion total entry num = 0.025441570 Billion write speed all = 58.589212250 MB/sec (with seq num) write speed all = 56.977184364 MB/sec (without seq num) 2024/11/19-15:20:43.227108 139851179493120 [dcompact_etcd.cc:1163] job-00138/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.286 sec, input = 126.908 MiB 2024/11/19-15:20:43.318309 139851148035840 [compaction_job.cc:1948] [default] [JOB 133] Generated table #4791: 63575 keys, 64975172 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:43.318367 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000843318344, "cf_name": "default", "job": 133, "event": "table_file_creation", "file_number": 4791, "file_size": 64975172, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64865328, "index_size": 108288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1271500, "raw_average_key_size": 20, "raw_value_size": 64658482, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63575, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000843, "oldest_key_time": 0, "file_creation_time": 1732000843, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4791, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:43.332777 139851148035840 [compaction_job.cc:703] job-00133: subcompact[0], size: 0.131113 G, files: /storage/mytopling/.rocksdb [004790,004791] 2024/11/19-15:20:43.333516 139851148035840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777215 2024/11/19-15:20:43.333647 139851148035840 (Original Log Time 2024/11/19-15:20:42.765686) EVENT_LOG_v1 {"time_micros": 1732000842765666, "job": 133, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4680, 4681, 4682, 4683], "score": 3.79302, "input_data_size": 134100552, "oldest_snapshot_seqno": 127414640} 2024/11/19-15:20:43.333653 139851148035840 (Original Log Time 2024/11/19-15:20:43.333227) [compaction_job.cc:2022] [default] [JOB 133] Compacted 4@1 files to L2 => 131113032 bytes 2024/11/19-15:20:43.333656 139851148035840 (Original Log Time 2024/11/19-15:20:43.333476) [compaction_job.cc:1224] [default] compacted to: files[5 55 53 16 0 0 0] max score 1.92 (89 files need compaction), MB/sec: 236.5 rd, 231.2 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128290, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:43.333672 139851148035840 (Original Log Time 2024/11/19-15:20:43.333508) EVENT_LOG_v1 {"time_micros": 1732000843333487, "cf": "default", "job": 133, "event": "compaction_finished", "compaction_time_micros": 567106, "compaction_time_cpu_micros": 406304, "output_level": 2, "num_output_files": 2, "total_output_size": 131113032, "num_input_records": 128290, "num_output_records": 128290, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 55, 53, 16, 0, 0, 0]} 2024/11/19-15:20:43.337968 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004661.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:43.337993 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000843337991, "job": 133, "event": "table_file_deletion", "file_number": 4661} 2024/11/19-15:20:43.342543 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004663.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:43.342564 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000843342561, "job": 133, "event": "table_file_deletion", "file_number": 4663} 2024/11/19-15:20:43.346650 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004665.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:43.346669 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000843346667, "job": 133, "event": "table_file_deletion", "file_number": 4665} 2024/11/19-15:20:43.350788 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004667.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:43.350807 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000843350805, "job": 133, "event": "table_file_deletion", "file_number": 4667} 2024/11/19-15:20:43.354703 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004675.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:43.354721 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000843354719, "job": 133, "event": "table_file_deletion", "file_number": 4675} 2024/11/19-15:20:43.354846 139851148035840 [compaction_job.cc:2437] [default] [JOB 143] Compacting 4@1 files to L2, score 1.92, subcompactions 1 : 1 2024/11/19-15:20:43.354855 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 141 Base level 1, inputs: [4711(31MB) 4712(31MB) 4713(31MB) 4714(31MB)] 2024/11/19-15:20:43.639544 139851148035840 [dcompact_etcd.cc:1163] job-00143/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.284 sec, input = 126.830 MiB 2024/11/19-15:20:44.626231 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-00 2024/11/19-15:20:44.626309 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00142/att-00/compact.done) = No such file or directory: 5.176 sec on http probe failed: 2, retry 2024/11/19-15:20:44.825567 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-01 2024/11/19-15:20:44.825675 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00141/att-01/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:20:44.921623 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-02 2024/11/19-15:20:44.921705 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00137/att-02/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:20:45.090958 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-02 2024/11/19-15:20:45.091050 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00136/att-02/compact.done) = No such file or directory: 9.206 sec on http probe failed: 4, retry 2024/11/19-15:20:45.197601 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-01 2024/11/19-15:20:45.197666 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00139/att-01/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:20:45.197683 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00139/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.850 sec, real 11.480 (submit 268.436 ms wait 11.202 s), input 126.850 MiB, send shutdown 2024/11/19-15:20:45.207415 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-01 2024/11/19-15:20:45.207641 139851169007360 [dcompact_etcd.cc:1773] job-00139/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:45.464194 139851169007360 [dcompact_etcd.cc:1163] job-00139/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.256 sec, input = 126.850 MiB 2024/11/19-15:20:46.408864 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-02 2024/11/19-15:20:46.408954 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00138/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:20:46.635932 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-00 2024/11/19-15:20:46.636005 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00142/att-00/compact.done) = No such file or directory: 7.185 sec on http probe failed: 3, retry 2024/11/19-15:20:46.818730 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-00 2024/11/19-15:20:46.818825 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00143/att-00/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:20:46.834178 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-01 2024/11/19-15:20:46.834239 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00141/att-01/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:20:46.930139 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-02 2024/11/19-15:20:46.930251 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00137/att-02/compact.done) = No such file or directory: 7.192 sec on http probe failed: 3, retry 2024/11/19-15:20:47.098872 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-02 2024/11/19-15:20:47.098957 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00136/att-02/compact.done) = No such file or directory: 11.215 sec on http probe failed: 5, giveup 2024/11/19-15:20:47.098976 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00136/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.987 sec, real 11.493 (submit 269.964 ms wait 11.215 s), input 126.987 MiB, send shutdown 2024/11/19-15:20:47.103841 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00136/att-02 2024/11/19-15:20:47.104068 139851137550080 [dcompact_etcd.cc:1773] job-00136/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:47.105942 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00136) = success 2024/11/19-15:20:47.105949 139851137550080 [dcompact_etcd.cc:1773] job-00136/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:20:47.105969 139851137550080 (Original Log Time 2024/11/19-15:20:12.598403) EVENT_LOG_v1 {"time_micros": 1732000812598397, "job": 136, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4686, 4688, 4689, 4690], "score": 3.41835, "input_data_size": 134100150, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:47.105976 139851137550080 [compaction_job.cc:2437] [default] [JOB 136] Compacting 4@1 files to L2, score 3.42, subcompactions 1 : 1 2024/11/19-15:20:47.105984 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 134 Base level 1, inputs: [4686(31MB) 4688(31MB) 4689(31MB) 4690(31MB)] 2024/11/19-15:20:47.106226 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004792.sst) /storage/mytopling/..//tmp/Topling-82038-I7zSsi 2024/11/19-15:20:47.159141 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004792.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:20:47.159274 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004792.sst): index pass time = 0.00's,5107.404'MB/sec index type = UintIndex_AllOne usrkeys = 64714 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:20:47.235570 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004792.sst): first pass time = 0.13's, 518.894'MB/sec 2024/11/19-15:20:47.235623 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004792.sst): old prealloc_size = 73819750, real_size = 66042200 2024/11/19-15:20:47.235646 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004792.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64714 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.02 avg-zval =1020.27 2024/11/19-15:20:47.290779 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004792.sst): second pass time = 0.00's,164950273.183'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1197.403'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.884'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.404'MB/sec write SST data time = 0.02's, 4368.157'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 164950273.183'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64714 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.02 avg-zval = 1020.27 usrkeys = 64714 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.02 avg-zval = 1020.27 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.6250 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 6.808632 GB avg = 0.267 KB (by entry num) total key len = 0.655875 GB avg = 0.026 KB total ukey len = 0.451825 GB avg = 0.018 KB total ukey num = 0.025506284 Billion total entry num = 0.025506284 Billion write speed all = 57.277151134 MB/sec (with seq num) write speed all = 55.711419185 MB/sec (without seq num) 2024/11/19-15:20:47.326961 139851137550080 [compaction_job.cc:1948] [default] [JOB 136] Generated table #4792: 64714 keys, 66027018 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:47.327021 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000847326997, "cf_name": "default", "job": 136, "event": "table_file_creation", "file_number": 4792, "file_size": 66027018, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025696, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294280, "raw_average_key_size": 20, "raw_value_size": 65815159, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64714, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000847, "oldest_key_time": 0, "file_creation_time": 1732000847, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4792, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:47.342332 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004793.sst) /storage/mytopling/..//tmp/Topling-82038-cKq7oc 2024/11/19-15:20:47.394321 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004793.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:20:47.397640 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004793.sst): index pass time = 0.00's, 155.179'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63690 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:20:47.460635 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004793.sst): first pass time = 0.12's, 558.385'MB/sec 2024/11/19-15:20:47.460690 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004793.sst): old prealloc_size = 73819750, real_size = 65104048 2024/11/19-15:20:47.460747 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004793.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63690 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0648 GB zip-val = 0.0650 GB avg-val =1016.99 avg-zval =1020.25 2024/11/19-15:20:47.460794 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004793.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.066 GB 2024/11/19-15:20:47.520046 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004793.sst): second pass time = 0.00's,104809519.417'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1096.889'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2045.596'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.372'MB/sec write SST data time = 0.02's, 3943.105'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 104809519.417'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63690 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.25 usrkeys = 63690 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7937 UnZipSize{ index = 0.0008 GB value = 0.0648 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0650 GB dict = 0.00 MB all = 0.0651 GB } UnZip/Zip{ index = 7.0454 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 6.873405 GB avg = 0.269 KB (by entry num) total key len = 0.657149 GB avg = 0.026 KB total ukey len = 0.452589 GB avg = 0.018 KB total ukey num = 0.025569974 Billion total entry num = 0.025569974 Billion write speed all = 57.682464352 MB/sec (with seq num) write speed all = 56.115579210 MB/sec (without seq num) 2024/11/19-15:20:47.538197 139851137550080 [compaction_job.cc:1948] [default] [JOB 136] Generated table #4793: 63690 keys, 65089540 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:47.538254 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000847538230, "cf_name": "default", "job": 136, "event": "table_file_creation", "file_number": 4793, "file_size": 65089540, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64979488, "index_size": 108480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1273800, "raw_average_key_size": 20, "raw_value_size": 64772283, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63690, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000847, "oldest_key_time": 0, "file_creation_time": 1732000847, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4793, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:47.553209 139851137550080 [compaction_job.cc:703] job-00136: subcompact[0], size: 0.131117 G, files: /storage/mytopling/.rocksdb [004792,004793] 2024/11/19-15:20:47.553884 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:20:47.554033 139851137550080 (Original Log Time 2024/11/19-15:20:47.106012) EVENT_LOG_v1 {"time_micros": 1732000847105996, "job": 136, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4686, 4688, 4689, 4690], "score": 3.41835, "input_data_size": 134100150, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:47.554040 139851137550080 (Original Log Time 2024/11/19-15:20:47.553601) [compaction_job.cc:2022] [default] [JOB 136] Compacted 4@1 files to L2 => 131116558 bytes 2024/11/19-15:20:47.554043 139851137550080 (Original Log Time 2024/11/19-15:20:47.553852) [compaction_job.cc:1224] [default] compacted to: files[5 51 55 16 0 0 0] max score 1.86 (87 files need compaction), MB/sec: 299.9 rd, 293.2 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128404, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:47.554062 139851137550080 (Original Log Time 2024/11/19-15:20:47.553877) EVENT_LOG_v1 {"time_micros": 1732000847553864, "cf": "default", "job": 136, "event": "compaction_finished", "compaction_time_micros": 447211, "compaction_time_cpu_micros": 395754, "output_level": 2, "num_output_files": 2, "total_output_size": 131116558, "num_input_records": 128404, "num_output_records": 128404, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 51, 55, 16, 0, 0, 0]} 2024/11/19-15:20:47.558337 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004646.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:47.558362 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000847558360, "job": 136, "event": "table_file_deletion", "file_number": 4646} 2024/11/19-15:20:47.562530 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004649.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:47.562553 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000847562551, "job": 136, "event": "table_file_deletion", "file_number": 4649} 2024/11/19-15:20:47.566674 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004654.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:47.566695 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000847566693, "job": 136, "event": "table_file_deletion", "file_number": 4654} 2024/11/19-15:20:47.571076 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004656.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:47.571097 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000847571095, "job": 136, "event": "table_file_deletion", "file_number": 4656} 2024/11/19-15:20:47.571238 139851137550080 [compaction_job.cc:2437] [default] [JOB 144] Compacting 1@2 files to L3, score 1.86, subcompactions 1 : 1 2024/11/19-15:20:47.571246 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 142 Base level 2, inputs: [4742(3587B)] 2024/11/19-15:20:47.857344 139851137550080 [dcompact_etcd.cc:1163] job-00144/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.286 sec, input = 2.137 KiB 2024/11/19-15:20:47.875517 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-00 2024/11/19-15:20:47.875577 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00144/att-00/compact.done) = No such file or directory: 0.006 sec on http probe failed: 1, retry 2024/11/19-15:20:47.880895 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-00 2024/11/19-15:20:47.880948 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00144/att-00/compact.done) = No such file or directory: 0.018 sec on http probe failed: 2, retry 2024/11/19-15:20:47.886042 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-00 2024/11/19-15:20:47.886090 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00144/att-00/compact.done) = No such file or directory: 0.024 sec on http probe failed: 3, retry 2024/11/19-15:20:47.893945 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-00 2024/11/19-15:20:47.893991 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00144/att-00/compact.done) = No such file or directory: 0.029 sec on http probe failed: 4, retry 2024/11/19-15:20:47.901583 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-00 2024/11/19-15:20:47.901634 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00144/att-00/compact.done) = No such file or directory: 0.037 sec on http probe failed: 5, giveup 2024/11/19-15:20:47.901648 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00144/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.330 (submit 285.661 ms wait 0.037 s), input 2.137 KiB, send shutdown 2024/11/19-15:20:47.909739 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-00 2024/11/19-15:20:47.909944 139851137550080 [dcompact_etcd.cc:1773] job-00144/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:48.160187 139851137550080 [dcompact_etcd.cc:1163] job-00144/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.250 sec, input = 2.137 KiB 2024/11/19-15:20:48.178547 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-01 2024/11/19-15:20:48.178612 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00144/att-01/compact.done) = No such file or directory: 0.009 sec on http probe failed: 1, retry 2024/11/19-15:20:48.186412 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-01 2024/11/19-15:20:48.186457 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00144/att-01/compact.done) = No such file or directory: 0.019 sec on http probe failed: 2, retry 2024/11/19-15:20:48.191881 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-01 2024/11/19-15:20:48.191934 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00144/att-01/compact.done) = No such file or directory: 0.026 sec on http probe failed: 3, retry 2024/11/19-15:20:48.199664 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-01 2024/11/19-15:20:48.199709 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00144/att-01/compact.done) = No such file or directory: 0.032 sec on http probe failed: 4, retry 2024/11/19-15:20:48.204805 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-01 2024/11/19-15:20:48.204851 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00144/att-01/compact.done) = No such file or directory: 0.040 sec on http probe failed: 5, giveup 2024/11/19-15:20:48.204865 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00144/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.295 (submit 250.172 ms wait 0.040 s), input 2.137 KiB, send shutdown 2024/11/19-15:20:48.213567 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-01 2024/11/19-15:20:48.213786 139851137550080 [dcompact_etcd.cc:1773] job-00144/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:48.417706 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-02 2024/11/19-15:20:48.417780 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00138/att-02/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:20:48.511417 139851137550080 [dcompact_etcd.cc:1163] job-00144/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.298 sec, input = 2.137 KiB 2024/11/19-15:20:48.516319 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-02 2024/11/19-15:20:48.516379 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00144/att-02/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:20:48.525118 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-02 2024/11/19-15:20:48.525165 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00144/att-02/compact.done) = No such file or directory: 0.005 sec on http probe failed: 2, retry 2024/11/19-15:20:48.534165 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-02 2024/11/19-15:20:48.534211 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00144/att-02/compact.done) = No such file or directory: 0.014 sec on http probe failed: 3, retry 2024/11/19-15:20:48.538745 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-02 2024/11/19-15:20:48.538788 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00144/att-02/compact.done) = No such file or directory: 0.023 sec on http probe failed: 4, retry 2024/11/19-15:20:48.547035 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-02 2024/11/19-15:20:48.547079 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00144/att-02/compact.done) = No such file or directory: 0.027 sec on http probe failed: 5, giveup 2024/11/19-15:20:48.547092 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00144/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.333 (submit 297.554 ms wait 0.027 s), input 2.137 KiB, send shutdown 2024/11/19-15:20:48.554096 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00144/att-02 2024/11/19-15:20:48.554297 139851137550080 [dcompact_etcd.cc:1773] job-00144/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:48.556076 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00144) = success 2024/11/19-15:20:48.556084 139851137550080 [dcompact_etcd.cc:1773] job-00144/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:20:48.556102 139851137550080 (Original Log Time 2024/11/19-15:20:47.571258) EVENT_LOG_v1 {"time_micros": 1732000847571251, "job": 144, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4742], "score": 1.85566, "input_data_size": 3587, "oldest_snapshot_seqno": 129844060} 2024/11/19-15:20:48.556109 139851137550080 [compaction_job.cc:2437] [default] [JOB 144] Compacting 1@2 files to L3, score 1.86, subcompactions 1 : 1 2024/11/19-15:20:48.556116 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 142 Base level 2, inputs: [4742(3587B)] 2024/11/19-15:20:48.556337 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004794.sst) /storage/mytopling/..//tmp/Topling-82038-HmXh1h 2024/11/19-15:20:48.556453 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004794.sst): first pass time = 0.00's, 20.444'MB/sec 2024/11/19-15:20:48.556473 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004794.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:20:48.556502 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004794.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, dictZip workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:20:48.556582 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004794.sst): index pass time = 0.00's, 0.776'MB/sec index type = UintIndex_AllOne usrkeys = 16 min-keylen = 4 max-keylen = 4 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 10.00 2024/11/19-15:20:48.576904 139851137550080 [top_zip_table_builder.cc:2752] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004794.sst): part-0: reject ZipStore ratio = 0.903 2024/11/19-15:20:48.577120 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004794.sst): old prealloc_size = 3945, real_size = 2576 2024/11/19-15:20:48.577137 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004794.sst): index type = UintIndex_AllOne , store type = ZipOffsetBlobStore usrkeys = 16 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 10.00 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 120.75 avg-zval = 138.00 2024/11/19-15:20:48.577281 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004794.sst): second pass time = 0.02's, 0.095'MB/sec, value only(88.3% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 23.067'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.697'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.385'MB/sec write SST data time = 0.00's, 28.114'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = 0.000'MB/sec zip my value time = 0.02's, unzip length = 0.000'GB zip my value throughput = 0.095'MB/sec zip pipeline throughput = 0.095'MB/sec entries = 16 avg-key = 16.00 avg-zkey = 12.00 avg-val = 120.75 avg-zval = 138.00 usrkeys = 16 avg-key = 8.00 avg-zkey = 12.00 avg-val = 120.75 avg-zval = 138.00 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 4 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.6667 value = 0.8750 dict = 0.00 all = 0.6225 } Zip/UnZip{ index = 1.5000 value = 1.1429 dict = 0.00 all = 1.6065 } ---------------------------- total value len = 6.873407 GB avg = 0.269 KB (by entry num) total key len = 0.657149 GB avg = 0.026 KB total ukey len = 0.452589 GB avg = 0.018 KB total ukey num = 0.025569990 Billion total entry num = 0.025569990 Billion write speed all = 57.219103799 MB/sec (with seq num) write speed all = 55.664804852 MB/sec (without seq num) 2024/11/19-15:20:48.577455 139851137550080 [compaction_job.cc:1948] [default] [JOB 144] Generated table #4794: 16 keys, 3515 bytes, temperature: kUnknown 2024/11/19-15:20:48.577493 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000848577471, "cf_name": "default", "job": 144, "event": "table_file_creation", "file_number": 4794, "file_size": 3515, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 2208, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 256, "raw_average_key_size": 16, "raw_value_size": 1932, "raw_average_value_size": 120, "num_data_blocks": 1, "num_entries": 16, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-ZipOffsetBlobStore", "creation_time": 1732000848, "oldest_key_time": 0, "file_creation_time": 1732000848, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4794, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:48.577580 139851137550080 [compaction_job.cc:703] job-00144: subcompact[0], size: 0.000004 G, files: /storage/mytopling/.rocksdb [004794] 2024/11/19-15:20:48.578022 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:20:48.578132 139851137550080 (Original Log Time 2024/11/19-15:20:48.556145) EVENT_LOG_v1 {"time_micros": 1732000848556129, "job": 144, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4742], "score": 1.85566, "input_data_size": 3587, "oldest_snapshot_seqno": 129844060} 2024/11/19-15:20:48.578137 139851137550080 (Original Log Time 2024/11/19-15:20:48.577770) [compaction_job.cc:2022] [default] [JOB 144] Compacted 1@2 files to L3 => 3515 bytes 2024/11/19-15:20:48.578140 139851137550080 (Original Log Time 2024/11/19-15:20:48.577998) [compaction_job.cc:1224] [default] compacted to: files[5 51 54 17 0 0 0] max score 1.86 (86 files need compaction), MB/sec: 0.2 rd, 0.2 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 16, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:48.578154 139851137550080 (Original Log Time 2024/11/19-15:20:48.578017) EVENT_LOG_v1 {"time_micros": 1732000848578006, "cf": "default", "job": 144, "event": "compaction_finished", "compaction_time_micros": 21440, "compaction_time_cpu_micros": 1250, "output_level": 3, "num_output_files": 1, "total_output_size": 3515, "num_input_records": 16, "num_output_records": 16, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 51, 54, 17, 0, 0, 0]} 2024/11/19-15:20:48.578212 139851137550080 [compaction_job.cc:2437] [default] [JOB 145] Compacting 1@2 files to L3, score 1.86, subcompactions 1 : 1 2024/11/19-15:20:48.578219 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 143 Base level 2, inputs: [4743(30KB)] 2024/11/19-15:20:48.641341 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-00 2024/11/19-15:20:48.641405 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00142/att-00/compact.done) = No such file or directory: 9.194 sec on http probe failed: 4, retry 2024/11/19-15:20:48.643232 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-02 2024/11/19-15:20:48.643293 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00139/att-02/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:20:48.827179 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-00 2024/11/19-15:20:48.827253 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00143/att-00/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:20:48.843435 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-01 2024/11/19-15:20:48.843494 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00141/att-01/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:20:48.844656 139851137550080 [dcompact_etcd.cc:1163] job-00145/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.266 sec, input = 22.213 KiB 2024/11/19-15:20:48.936768 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-02 2024/11/19-15:20:48.936823 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00137/att-02/compact.done) = No such file or directory: 9.200 sec on http probe failed: 4, retry 2024/11/19-15:20:48.944726 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-00 2024/11/19-15:20:48.944783 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00145/att-00/compact.done) = No such file or directory: 0.092 sec on http probe failed: 1, retry 2024/11/19-15:20:48.949741 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-00 2024/11/19-15:20:48.949787 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00145/att-00/compact.done) = No such file or directory: 0.101 sec on http probe failed: 2, retry 2024/11/19-15:20:48.958178 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-00 2024/11/19-15:20:48.958222 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00145/att-00/compact.done) = No such file or directory: 0.106 sec on http probe failed: 3, retry 2024/11/19-15:20:48.966713 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-00 2024/11/19-15:20:48.966757 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00145/att-00/compact.done) = No such file or directory: 0.114 sec on http probe failed: 4, retry 2024/11/19-15:20:48.974684 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-00 2024/11/19-15:20:48.974729 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00145/att-00/compact.done) = No such file or directory: 0.123 sec on http probe failed: 5, giveup 2024/11/19-15:20:48.974743 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00145/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.396 (submit 266.049 ms wait 0.123 s), input 22.213 KiB, send shutdown 2024/11/19-15:20:48.983250 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-00 2024/11/19-15:20:48.983469 139851137550080 [dcompact_etcd.cc:1773] job-00145/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:49.244452 139851137550080 [dcompact_etcd.cc:1163] job-00145/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.261 sec, input = 22.213 KiB 2024/11/19-15:20:49.330363 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-01 2024/11/19-15:20:49.330440 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00145/att-01/compact.done) = No such file or directory: 0.078 sec on http probe failed: 1, retry 2024/11/19-15:20:49.335690 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-01 2024/11/19-15:20:49.335735 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00145/att-01/compact.done) = No such file or directory: 0.086 sec on http probe failed: 2, retry 2024/11/19-15:20:49.341158 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-01 2024/11/19-15:20:49.341206 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00145/att-01/compact.done) = No such file or directory: 0.092 sec on http probe failed: 3, retry 2024/11/19-15:20:49.350026 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-01 2024/11/19-15:20:49.350083 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00145/att-01/compact.done) = No such file or directory: 0.097 sec on http probe failed: 4, retry 2024/11/19-15:20:49.358941 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-01 2024/11/19-15:20:49.358986 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00145/att-01/compact.done) = No such file or directory: 0.106 sec on http probe failed: 5, giveup 2024/11/19-15:20:49.359001 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00145/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.375 (submit 260.902 ms wait 0.106 s), input 22.213 KiB, send shutdown 2024/11/19-15:20:49.363400 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-01 2024/11/19-15:20:49.363606 139851137550080 [dcompact_etcd.cc:1773] job-00145/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:49.641103 139851137550080 [dcompact_etcd.cc:1163] job-00145/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.277 sec, input = 22.213 KiB 2024/11/19-15:20:49.733154 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-02 2024/11/19-15:20:49.733224 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00145/att-02/compact.done) = No such file or directory: 0.087 sec on http probe failed: 1, retry 2024/11/19-15:20:49.738244 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-02 2024/11/19-15:20:49.738291 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00145/att-02/compact.done) = No such file or directory: 0.093 sec on http probe failed: 2, retry 2024/11/19-15:20:49.743504 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-02 2024/11/19-15:20:49.743549 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00145/att-02/compact.done) = No such file or directory: 0.098 sec on http probe failed: 3, retry 2024/11/19-15:20:49.753127 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-02 2024/11/19-15:20:49.753174 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00145/att-02/compact.done) = No such file or directory: 0.103 sec on http probe failed: 4, retry 2024/11/19-15:20:49.761084 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-02 2024/11/19-15:20:49.761128 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00145/att-02/compact.done) = No such file or directory: 0.113 sec on http probe failed: 5, giveup 2024/11/19-15:20:49.761143 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00145/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.397 (submit 277.423 ms wait 0.113 s), input 22.213 KiB, send shutdown 2024/11/19-15:20:49.766462 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00145/att-02 2024/11/19-15:20:49.766665 139851137550080 [dcompact_etcd.cc:1773] job-00145/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:49.768467 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00145) = success 2024/11/19-15:20:49.768474 139851137550080 [dcompact_etcd.cc:1773] job-00145/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:20:49.768491 139851137550080 (Original Log Time 2024/11/19-15:20:48.578229) EVENT_LOG_v1 {"time_micros": 1732000848578224, "job": 145, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4743], "score": 1.85565, "input_data_size": 31440, "oldest_snapshot_seqno": 129884896} 2024/11/19-15:20:49.768498 139851137550080 [compaction_job.cc:2437] [default] [JOB 145] Compacting 1@2 files to L3, score 1.86, subcompactions 1 : 1 2024/11/19-15:20:49.768505 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 143 Base level 2, inputs: [4743(30KB)] 2024/11/19-15:20:49.768767 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004795.sst) /storage/mytopling/..//tmp/Topling-82038-Nttk7G 2024/11/19-15:20:49.768984 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004795.sst): first pass time = 0.00's, 109.266'MB/sec 2024/11/19-15:20:49.768992 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004795.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:20:49.769018 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004795.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, dictZip workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:20:49.769156 139834066732800 [fixed_len_key_index.cc:363] FixedLenKeyIndex build_cache Finish: num_keys = 161, fixlen = 8, pref_len = 4, cplen = 3, time = 0.000 sec 2024/11/19-15:20:49.769240 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004795.sst): index pass time = 0.00's, 5.728'MB/sec index type = FixedLenKeyIndex usrkeys = 161 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 60.42 2024/11/19-15:20:49.789712 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004795.sst): old prealloc_size = 34584, real_size = 23536 2024/11/19-15:20:49.789746 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004795.sst): index type = FixedLenKeyIndex , store type = DictZipBlobStore usrkeys = 161 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 60.42 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 121.28 avg-zval = 84.47 2024/11/19-15:20:49.790016 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004795.sst): second pass time = 0.02's, 0.944'MB/sec, value only(85.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 89.625'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 356.755'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 3.702'MB/sec write SST data time = 0.00's, 103.481'MB/sec dict compress time = 0.00's, 3.183'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = 2.824'MB/sec zip my value time = 0.02's, unzip length = 0.000'GB zip my value throughput = 0.944'MB/sec zip pipeline throughput = 0.944'MB/sec entries = 161 avg-key = 20.00 avg-zkey = 60.42 avg-val = 121.28 avg-zval = 84.47 usrkeys = 161 avg-key = 12.00 avg-zkey = 60.42 avg-val = 121.28 avg-zval = 84.47 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 3 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.1986 value = 1.4357 dict = 1.00 all = 0.9150 } Zip/UnZip{ index = 5.0352 value = 0.6965 dict = 1.00 all = 1.0929 } ---------------------------- total value len = 6.873426 GB avg = 0.269 KB (by entry num) total key len = 0.657153 GB avg = 0.026 KB total ukey len = 0.452591 GB avg = 0.018 KB total ukey num = 0.025570151 Billion total entry num = 0.025570151 Billion write speed all = 56.696842429 MB/sec (with seq num) write speed all = 55.156725137 MB/sec (without seq num) 2024/11/19-15:20:49.790265 139851137550080 [compaction_job.cc:1948] [default] [JOB 145] Generated table #4795: 161 keys, 24859 bytes, temperature: kUnknown 2024/11/19-15:20:49.790319 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000849790288, "cf_name": "default", "job": 145, "event": "table_file_creation", "file_number": 4795, "file_size": 24859, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 13600, "index_size": 9728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3220, "raw_average_key_size": 20, "raw_value_size": 19526, "raw_average_value_size": 121, "num_data_blocks": 1, "num_entries": 161, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-FixedLenKeyIndex-DictZipBlobStore", "creation_time": 1732000849, "oldest_key_time": 0, "file_creation_time": 1732000849, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4795, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:49.790463 139851137550080 [compaction_job.cc:703] job-00145: subcompact[0], size: 0.000025 G, files: /storage/mytopling/.rocksdb [004795] 2024/11/19-15:20:49.791072 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:20:49.791209 139851137550080 (Original Log Time 2024/11/19-15:20:49.768534) EVENT_LOG_v1 {"time_micros": 1732000849768518, "job": 145, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4743], "score": 1.85565, "input_data_size": 31440, "oldest_snapshot_seqno": 129884896} 2024/11/19-15:20:49.791218 139851137550080 (Original Log Time 2024/11/19-15:20:49.790724) [compaction_job.cc:2022] [default] [JOB 145] Compacted 1@2 files to L3 => 24859 bytes 2024/11/19-15:20:49.791223 139851137550080 (Original Log Time 2024/11/19-15:20:49.791040) [compaction_job.cc:1224] [default] compacted to: files[5 51 53 18 0 0 0] max score 1.86 (85 files need compaction), MB/sec: 1.4 rd, 1.1 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 161, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:49.791244 139851137550080 (Original Log Time 2024/11/19-15:20:49.791064) EVENT_LOG_v1 {"time_micros": 1732000849791051, "cf": "default", "job": 145, "event": "compaction_finished", "compaction_time_micros": 21938, "compaction_time_cpu_micros": 1717, "output_level": 3, "num_output_files": 1, "total_output_size": 24859, "num_input_records": 161, "num_output_records": 161, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 51, 53, 18, 0, 0, 0]} 2024/11/19-15:20:49.791332 139851137550080 [compaction_job.cc:2437] [default] [JOB 146] Compacting 4@2 files to L3, score 1.86, subcompactions 1 : 1 2024/11/19-15:20:49.791345 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 144 Base level 2, inputs: [4763(62MB) 4764(61MB) 4765(62MB) 4766(61MB)] 2024/11/19-15:20:50.068178 139851137550080 [dcompact_etcd.cc:1163] job-00146/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.276 sec, input = 254.479 MiB 2024/11/19-15:20:50.426330 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-02 2024/11/19-15:20:50.426416 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00138/att-02/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:20:50.650286 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-00 2024/11/19-15:20:50.650382 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00142/att-00/compact.done) = No such file or directory: 11.200 sec on http probe failed: 5, giveup 2024/11/19-15:20:50.650401 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00142/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.813 sec, real 11.533 (submit 324.020 ms wait 11.200 s), input 126.813 MiB, send shutdown 2024/11/19-15:20:50.652323 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-02 2024/11/19-15:20:50.652385 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00139/att-02/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:20:50.655447 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-00 2024/11/19-15:20:50.655671 139851127064320 [dcompact_etcd.cc:1773] job-00142/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:50.836884 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-00 2024/11/19-15:20:50.836971 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00143/att-00/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:20:50.849042 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-01 2024/11/19-15:20:50.849098 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00141/att-01/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:20:50.945585 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-02 2024/11/19-15:20:50.945644 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00137/att-02/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:20:50.945660 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00137/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.986 sec, real 11.486 (submit 270.348 ms wait 11.207 s), input 126.986 MiB, send shutdown 2024/11/19-15:20:50.949901 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00137/att-02 2024/11/19-15:20:50.955719 139851127064320 [dcompact_etcd.cc:1163] job-00142/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.300 sec, input = 126.813 MiB 2024/11/19-15:20:50.994871 139851158521600 [dcompact_etcd.cc:1773] job-00137/att-02: CleanFiles 0.045 sec : 0.000 2024/11/19-15:20:51.005032 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00137) = success 2024/11/19-15:20:51.005042 139851158521600 [dcompact_etcd.cc:1773] job-00137/att-03: CleanFiles 0.010 sec : 0.000 2024/11/19-15:20:51.005063 139851158521600 (Original Log Time 2024/11/19-15:20:16.470668) EVENT_LOG_v1 {"time_micros": 1732000816470660, "job": 137, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4691, 4692, 4693, 4694], "score": 3.16857, "input_data_size": 134098688, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:51.005070 139851158521600 [compaction_job.cc:2437] [default] [JOB 137] Compacting 4@1 files to L2, score 3.17, subcompactions 1 : 1 2024/11/19-15:20:51.005077 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 135 Base level 1, inputs: [4691(31MB) 4692(31MB) 4693(31MB) 4694(31MB)] 2024/11/19-15:20:51.005361 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004796.sst) /storage/mytopling/..//tmp/Topling-82038-q7T2bh 2024/11/19-15:20:51.057793 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004796.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:20:51.057951 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004796.sst): index pass time = 0.00's,4080.041'MB/sec index type = UintIndex_AllOne usrkeys = 64712 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:20:51.124305 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004796.sst): first pass time = 0.12's, 564.287'MB/sec 2024/11/19-15:20:51.124367 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004796.sst): old prealloc_size = 73819750, real_size = 66041832 2024/11/19-15:20:51.124392 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004796.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64712 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.04 avg-zval =1020.29 2024/11/19-15:20:51.178291 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004796.sst): second pass time = 0.00's,132424128.773'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1224.600'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.514'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.419'MB/sec write SST data time = 0.02's, 4314.184'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 132424128.773'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64712 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.04 avg-zval = 1020.29 usrkeys = 64712 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.04 avg-zval = 1020.29 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.5000 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 6.939241 GB avg = 0.271 KB (by entry num) total key len = 0.658447 GB avg = 0.026 KB total ukey len = 0.453368 GB avg = 0.018 KB total ukey num = 0.025634863 Billion total entry num = 0.025634863 Billion write speed all = 56.610394396 MB/sec (with seq num) write speed all = 55.082350872 MB/sec (without seq num) 2024/11/19-15:20:51.181881 139851158521600 [compaction_job.cc:1948] [default] [JOB 137] Generated table #4796: 64712 keys, 66026634 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:51.181944 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000851181910, "cf_name": "default", "job": 137, "event": "table_file_creation", "file_number": 4796, "file_size": 66026634, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025328, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294240, "raw_average_key_size": 20, "raw_value_size": 65814792, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64712, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000851, "oldest_key_time": 0, "file_creation_time": 1732000851, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4796, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:51.196221 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004797.sst) /storage/mytopling/..//tmp/Topling-82038-n5Ppqv 2024/11/19-15:20:51.247488 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004797.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:20:51.250779 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004797.sst): index pass time = 0.00's, 156.415'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63690 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:20:51.312138 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004797.sst): first pass time = 0.12's, 569.845'MB/sec 2024/11/19-15:20:51.312190 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004797.sst): old prealloc_size = 73819750, real_size = 65103312 2024/11/19-15:20:51.312247 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004797.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63690 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0648 GB zip-val = 0.0650 GB avg-val =1016.98 avg-zval =1020.23 2024/11/19-15:20:51.312325 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004797.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.065 sec, Key+Value = 0.066 GB 2024/11/19-15:20:51.372024 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004797.sst): second pass time = 0.00's,151335371.495'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1088.112'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2025.354'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.370'MB/sec write SST data time = 0.02's, 3895.510'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 151335371.495'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63690 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 1020.23 usrkeys = 63690 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 1020.23 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7935 UnZipSize{ index = 0.0008 GB value = 0.0648 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0650 GB dict = 0.00 MB all = 0.0651 GB } UnZip/Zip{ index = 7.0454 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 7.004013 GB avg = 0.273 KB (by entry num) total key len = 0.659721 GB avg = 0.026 KB total ukey len = 0.454132 GB avg = 0.018 KB total ukey num = 0.025698553 Billion total entry num = 0.025698553 Billion write speed all = 57.020189424 MB/sec (with seq num) write speed all = 55.490557498 MB/sec (without seq num) 2024/11/19-15:20:51.442339 139851158521600 [compaction_job.cc:1948] [default] [JOB 137] Generated table #4797: 63690 keys, 65088772 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:51.442400 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000851442376, "cf_name": "default", "job": 137, "event": "table_file_creation", "file_number": 4797, "file_size": 65088772, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64978752, "index_size": 108480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1273800, "raw_average_key_size": 20, "raw_value_size": 64771539, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63690, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000851, "oldest_key_time": 0, "file_creation_time": 1732000851, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4797, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:51.456364 139851158521600 [compaction_job.cc:703] job-00137: subcompact[0], size: 0.131115 G, files: /storage/mytopling/.rocksdb [004796,004797] 2024/11/19-15:20:51.457022 139851158521600 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:20:51.457166 139851158521600 (Original Log Time 2024/11/19-15:20:51.005107) EVENT_LOG_v1 {"time_micros": 1732000851005089, "job": 137, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4691, 4692, 4693, 4694], "score": 3.16857, "input_data_size": 134098688, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:51.457176 139851158521600 (Original Log Time 2024/11/19-15:20:51.456734) [compaction_job.cc:2022] [default] [JOB 137] Compacted 4@1 files to L2 => 131115406 bytes 2024/11/19-15:20:51.457181 139851158521600 (Original Log Time 2024/11/19-15:20:51.456991) [compaction_job.cc:1224] [default] compacted to: files[5 47 55 18 0 0 0] max score 1.77 (83 files need compaction), MB/sec: 297.2 rd, 290.5 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128402, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:51.457204 139851158521600 (Original Log Time 2024/11/19-15:20:51.457016) EVENT_LOG_v1 {"time_micros": 1732000851457002, "cf": "default", "job": 137, "event": "compaction_finished", "compaction_time_micros": 451270, "compaction_time_cpu_micros": 382854, "output_level": 2, "num_output_files": 2, "total_output_size": 131115406, "num_input_records": 128402, "num_output_records": 128402, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 47, 55, 18, 0, 0, 0]} 2024/11/19-15:20:51.490241 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004658.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:51.490268 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000851490265, "job": 137, "event": "table_file_deletion", "file_number": 4658} 2024/11/19-15:20:51.494283 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004660.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:51.494305 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000851494303, "job": 137, "event": "table_file_deletion", "file_number": 4660} 2024/11/19-15:20:51.498629 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004662.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:51.498649 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000851498647, "job": 137, "event": "table_file_deletion", "file_number": 4662} 2024/11/19-15:20:51.502908 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004664.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:51.502934 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000851502932, "job": 137, "event": "table_file_deletion", "file_number": 4664} 2024/11/19-15:20:51.503067 139851158521600 [compaction_job.cc:2437] [default] [JOB 147] Compacting 1@2 files to L3, score 1.77, subcompactions 1 : 1 2024/11/19-15:20:51.503075 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 145 Base level 2, inputs: [4761(52MB)] 2024/11/19-15:20:51.820280 139851158521600 [dcompact_etcd.cc:1163] job-00147/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.317 sec, input = 53.697 MiB 2024/11/19-15:20:52.435386 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-02 2024/11/19-15:20:52.435489 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00138/att-02/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:20:52.661282 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-02 2024/11/19-15:20:52.661376 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00139/att-02/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:20:52.845894 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-00 2024/11/19-15:20:52.846002 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00143/att-00/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:20:52.854297 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-01 2024/11/19-15:20:52.854384 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00141/att-01/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:20:52.854403 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00141/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.972 sec, real 11.483 (submit 273.525 ms wait 11.204 s), input 126.972 MiB, send shutdown 2024/11/19-15:20:52.862104 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-01 2024/11/19-15:20:52.862357 139850978166528 [dcompact_etcd.cc:1773] job-00141/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:53.132863 139850978166528 [dcompact_etcd.cc:1163] job-00141/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.270 sec, input = 126.972 MiB 2024/11/19-15:20:53.171951 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-00 2024/11/19-15:20:53.172052 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00147/att-00/compact.done) = No such file or directory: 1.343 sec on http probe failed: 1, retry 2024/11/19-15:20:54.032769 139852349699840 [db_impl_write.cc:2181] [default] New memtable created with log file: #4798. Immutable memtables: 0. 2024/11/19-15:20:54.032964 139852349699840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16273899 2024/11/19-15:20:54.033246 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 148] Syncing log #4785 2024/11/19-15:20:54.033366 139850967680768 (Original Log Time 2024/11/19-15:20:54.033045) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:20:54.033379 139850967680768 [flush_job.cc:879] [default] [JOB 148] Flushing memtable with next log file: 4798 2024/11/19-15:20:54.033405 139850967680768 EVENT_LOG_v1 {"time_micros": 1732000854033399, "job": 148, "event": "flush_started", "num_memtables": 1, "num_entries": 1372698, "num_deletes": 0, "total_data_size": 459025076, "memory_usage": 538705288, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:20:54.033410 139850967680768 [flush_job.cc:910] [default] [JOB 148] Level-0 flush table #4799: started 2024/11/19-15:20:54.034031 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000017.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004799.sst in ConvertToSST, mem_size = 513.749 M, time = 611.048 us 2024/11/19-15:20:54.132725 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-01 2024/11/19-15:20:54.132888 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00142/att-01/compact.done) = No such file or directory: 3.170 sec on http probe failed: 1, retry 2024/11/19-15:20:54.255582 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-00 2024/11/19-15:20:54.255731 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00147/att-00/compact.done) = No such file or directory: 2.426 sec on http probe failed: 2, retry 2024/11/19-15:20:54.444404 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-02 2024/11/19-15:20:54.444488 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00138/att-02/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:20:54.444505 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00138/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.908 sec, real 11.503 (submit 285.510 ms wait 11.208 s), input 126.908 MiB, send shutdown 2024/11/19-15:20:54.453347 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00138/att-02 2024/11/19-15:20:54.478517 139851179493120 [dcompact_etcd.cc:1773] job-00138/att-02: CleanFiles 0.025 sec : 0.000 2024/11/19-15:20:54.486153 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00138) = success 2024/11/19-15:20:54.486168 139851179493120 [dcompact_etcd.cc:1773] job-00138/att-03: CleanFiles 0.008 sec : 0.000 2024/11/19-15:20:54.486162 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004799.sst): fsize = 513.751 M, time(ms): open: 0.035, seek: 0.015, finish: 0.039, meta: 0.013, Flush: 0.014, sync: 451.989, close: 0.010, all: 452.115 2024/11/19-15:20:54.486193 139851179493120 (Original Log Time 2024/11/19-15:20:19.850176) EVENT_LOG_v1 {"time_micros": 1732000819850169, "job": 138, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4695, 4696, 4697, 4698], "score": 2.91879, "input_data_size": 134098752, "oldest_snapshot_seqno": 128265508} 2024/11/19-15:20:54.486203 139851179493120 [compaction_job.cc:2437] [default] [JOB 138] Compacting 4@1 files to L2, score 2.92, subcompactions 1 : 1 2024/11/19-15:20:54.486214 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 136 Base level 1, inputs: [4695(31MB) 4696(31MB) 4697(31MB) 4698(31MB)] 2024/11/19-15:20:54.486473 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004800.sst) /storage/mytopling/..//tmp/Topling-82038-eQnV8U 2024/11/19-15:20:54.487554 139850967680768 [flush_job.cc:1083] [default] [JOB 148] Flush lasted 454189 microseconds, and 328250 cpu microseconds. 2024/11/19-15:20:54.518024 139850967680768 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 15785682 2024/11/19-15:20:54.518125 139850967680768 (Original Log Time 2024/11/19-15:20:54.486203) [flush_job.cc:1029] [default] [JOB 148] Level-0 flush table #4799: 538706714 bytes OK (needs compaction) 2024/11/19-15:20:54.518134 139850967680768 (Original Log Time 2024/11/19-15:20:54.487573) [memtable_list.cc:560] [default] Level-0 commit table #4799 started 2024/11/19-15:20:54.518140 139850967680768 (Original Log Time 2024/11/19-15:20:54.517980) [memtable_list.cc:759] [default] Level-0 commit table #4799: memtable #1 done 2024/11/19-15:20:54.518146 139850967680768 (Original Log Time 2024/11/19-15:20:54.518015) EVENT_LOG_v1 {"time_micros": 1732000854518007, "job": 148, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [6, 47, 55, 18, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:20:54.518151 139850967680768 (Original Log Time 2024/11/19-15:20:54.518090) [db_impl_compaction_flush.cc:310] [default] Level summary: files[6 47 55 18 0 0 0] max score 2.07 (83 files need compaction) 2024/11/19-15:20:54.551522 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004800.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:20:54.554839 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004800.sst): index pass time = 0.00's, 157.369'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64714 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:20:54.618091 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004800.sst): first pass time = 0.13's, 509.944'MB/sec 2024/11/19-15:20:54.618145 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004800.sst): old prealloc_size = 73819750, real_size = 66152480 2024/11/19-15:20:54.618202 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004800.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64714 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.02 avg-zval =1020.27 2024/11/19-15:20:54.618251 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004800.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.067 GB 2024/11/19-15:20:54.670150 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-02 2024/11/19-15:20:54.670236 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00139/att-02/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:20:54.679167 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004800.sst): second pass time = 0.00's,99720292.424'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1084.031'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2035.837'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.373'MB/sec write SST data time = 0.02's, 3760.696'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 99720292.424'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64714 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.02 avg-zval = 1020.27 usrkeys = 64714 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.02 avg-zval = 1020.27 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0464 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 7.069828 GB avg = 0.274 KB (by entry num) total key len = 0.661015 GB avg = 0.026 KB total ukey len = 0.454909 GB avg = 0.018 KB total ukey num = 0.025763267 Billion total entry num = 0.025763267 Billion write speed all = 56.138164742 MB/sec (with seq num) write speed all = 54.641507715 MB/sec (without seq num) 2024/11/19-15:20:54.682816 139851179493120 [compaction_job.cc:1948] [default] [JOB 138] Generated table #4800: 64714 keys, 66137732 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:54.682871 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000854682847, "cf_name": "default", "job": 138, "event": "table_file_creation", "file_number": 4800, "file_size": 66137732, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025936, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294280, "raw_average_key_size": 20, "raw_value_size": 65815393, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64714, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000854, "oldest_key_time": 0, "file_creation_time": 1732000854, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4800, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:54.698633 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004801.sst) /storage/mytopling/..//tmp/Topling-82038-28jxfo 2024/11/19-15:20:54.751637 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004801.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:20:54.751754 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004801.sst): index pass time = 0.00's,6104.465'MB/sec index type = UintIndex_AllOne usrkeys = 63607 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:20:54.818191 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004801.sst): first pass time = 0.12's, 551.808'MB/sec 2024/11/19-15:20:54.818244 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004801.sst): old prealloc_size = 73819750, real_size = 64913640 2024/11/19-15:20:54.818266 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004801.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63607 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1017.03 avg-zval =1020.29 2024/11/19-15:20:54.855234 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-00 2024/11/19-15:20:54.855343 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00143/att-00/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:20:54.855362 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00143/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.830 sec, real 11.500 (submit 284.144 ms wait 11.207 s), input 126.830 MiB, send shutdown 2024/11/19-15:20:54.865620 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-00 2024/11/19-15:20:54.865853 139851148035840 [dcompact_etcd.cc:1773] job-00143/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:54.872632 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004801.sst): second pass time = 0.00's,168904569.191'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1193.491'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.157'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.412'MB/sec write SST data time = 0.02's, 4107.819'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 168904569.191'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63607 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.03 avg-zval = 1020.29 usrkeys = 63607 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.03 avg-zval = 1020.29 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0000 GB value = 0.0649 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index =3975.4375 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 7.134518 GB avg = 0.276 KB (by entry num) total key len = 0.662287 GB avg = 0.026 KB total ukey len = 0.455672 GB avg = 0.018 KB total ukey num = 0.025826874 Billion total entry num = 0.025826874 Billion write speed all = 56.537729510 MB/sec (with seq num) write speed all = 55.039482224 MB/sec (without seq num) 2024/11/19-15:20:54.876313 139851179493120 [compaction_job.cc:1948] [default] [JOB 138] Generated table #4801: 63607 keys, 64898698 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:54.876364 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000854876339, "cf_name": "default", "job": 138, "event": "table_file_creation", "file_number": 4801, "file_size": 64898698, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64897392, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1272140, "raw_average_key_size": 20, "raw_value_size": 64690450, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63607, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000854, "oldest_key_time": 0, "file_creation_time": 1732000854, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4801, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:54.891652 139851179493120 [compaction_job.cc:703] job-00138: subcompact[0], size: 0.131036 G, files: /storage/mytopling/.rocksdb [004800,004801] 2024/11/19-15:20:54.892345 139851179493120 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16273898 2024/11/19-15:20:54.892497 139851179493120 (Original Log Time 2024/11/19-15:20:54.486252) EVENT_LOG_v1 {"time_micros": 1732000854486229, "job": 138, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4695, 4696, 4697, 4698], "score": 2.91879, "input_data_size": 134098752, "oldest_snapshot_seqno": 128265508} 2024/11/19-15:20:54.892507 139851179493120 (Original Log Time 2024/11/19-15:20:54.892061) [compaction_job.cc:2022] [default] [JOB 138] Compacted 4@1 files to L2 => 131036430 bytes 2024/11/19-15:20:54.892511 139851179493120 (Original Log Time 2024/11/19-15:20:54.892310) [compaction_job.cc:1224] [default] compacted to: files[6 43 57 18 0 0 0] max score 2.27 (85 files need compaction), MB/sec: 330.8 rd, 323.2 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128321, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:54.892533 139851179493120 (Original Log Time 2024/11/19-15:20:54.892337) EVENT_LOG_v1 {"time_micros": 1732000854892323, "cf": "default", "job": 138, "event": "compaction_finished", "compaction_time_micros": 405411, "compaction_time_cpu_micros": 403650, "output_level": 2, "num_output_files": 2, "total_output_size": 131036430, "num_input_records": 128321, "num_output_records": 128321, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 43, 57, 18, 0, 0, 0]} 2024/11/19-15:20:54.896684 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004669.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:54.896709 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000854896706, "job": 138, "event": "table_file_deletion", "file_number": 4669} 2024/11/19-15:20:54.900813 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004671.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:54.900834 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000854900832, "job": 138, "event": "table_file_deletion", "file_number": 4671} 2024/11/19-15:20:54.904852 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004672.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:54.904872 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000854904870, "job": 138, "event": "table_file_deletion", "file_number": 4672} 2024/11/19-15:20:54.908930 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004674.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:54.908949 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000854908947, "job": 138, "event": "table_file_deletion", "file_number": 4674} 2024/11/19-15:20:54.909095 139851179493120 [compaction_job.cc:2437] [default] [JOB 149] Compacting 1@2 files to L3, score 1.82, subcompactions 1 : 1 2024/11/19-15:20:54.909103 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 147 Base level 2, inputs: [4757(26MB)] 2024/11/19-15:20:55.073388 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-00 2024/11/19-15:20:55.073475 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00146/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:20:55.120509 139851148035840 [dcompact_etcd.cc:1163] job-00143/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.255 sec, input = 126.830 MiB 2024/11/19-15:20:55.199592 139851179493120 [dcompact_etcd.cc:1163] job-00149/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.290 sec, input = 33.874 MiB 2024/11/19-15:20:55.334747 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-00 2024/11/19-15:20:55.334821 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00147/att-00/compact.done) = No such file or directory: 3.509 sec on http probe failed: 3, retry 2024/11/19-15:20:56.052816 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-00 2024/11/19-15:20:56.052901 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00149/att-00/compact.done) = No such file or directory: 0.847 sec on http probe failed: 1, retry 2024/11/19-15:20:56.142151 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-01 2024/11/19-15:20:56.142227 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00142/att-01/compact.done) = No such file or directory: 5.177 sec on http probe failed: 2, retry 2024/11/19-15:20:56.316190 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-02 2024/11/19-15:20:56.316288 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00141/att-02/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:20:56.417334 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-00 2024/11/19-15:20:56.417428 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00147/att-00/compact.done) = No such file or directory: 4.589 sec on http probe failed: 4, retry 2024/11/19-15:20:56.679328 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-02 2024/11/19-15:20:56.679434 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00139/att-02/compact.done) = No such file or directory: 11.206 sec on http probe failed: 5, giveup 2024/11/19-15:20:56.679452 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00139/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.850 sec, real 11.472 (submit 256.470 ms wait 11.206 s), input 126.850 MiB, send shutdown 2024/11/19-15:20:56.688266 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00139/att-02 2024/11/19-15:20:56.736057 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-00 2024/11/19-15:20:56.736117 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00149/att-00/compact.done) = No such file or directory: 1.531 sec on http probe failed: 2, retry 2024/11/19-15:20:56.790067 139851169007360 [dcompact_etcd.cc:1773] job-00139/att-02: CleanFiles 0.102 sec : 0.000 2024/11/19-15:20:56.792171 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00139) = success 2024/11/19-15:20:56.792187 139851169007360 [dcompact_etcd.cc:1773] job-00139/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:20:56.792211 139851169007360 (Original Log Time 2024/11/19-15:20:22.177558) EVENT_LOG_v1 {"time_micros": 1732000822177550, "job": 139, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4699, 4700, 4701, 4702], "score": 2.66901, "input_data_size": 134100352, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:56.792219 139851169007360 [compaction_job.cc:2437] [default] [JOB 139] Compacting 4@1 files to L2, score 2.67, subcompactions 1 : 1 2024/11/19-15:20:56.792226 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 137 Base level 1, inputs: [4699(31MB) 4700(31MB) 4701(31MB) 4702(31MB)] 2024/11/19-15:20:56.792466 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004802.sst) /storage/mytopling/..//tmp/Topling-82038-Lc7tZ6 2024/11/19-15:20:56.845110 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004802.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:20:56.848482 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004802.sst): index pass time = 0.00's, 155.051'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64714 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:20:56.920907 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004802.sst): first pass time = 0.13's, 522.542'MB/sec 2024/11/19-15:20:56.920970 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004802.sst): old prealloc_size = 73819750, real_size = 66151744 2024/11/19-15:20:56.921029 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004802.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64714 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:20:56.921087 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004802.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.076 sec, Key+Value = 0.067 GB 2024/11/19-15:20:56.987187 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004802.sst): second pass time = 0.00's,190215800.578'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 998.978'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2027.746'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.328'MB/sec write SST data time = 0.02's, 3921.549'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 190215800.578'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64714 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 64714 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0464 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 7.200333 GB avg = 0.278 KB (by entry num) total key len = 0.663581 GB avg = 0.026 KB total ukey len = 0.456449 GB avg = 0.018 KB total ukey num = 0.025891588 Billion total entry num = 0.025891588 Billion write speed all = 56.163187548 MB/sec (with seq num) write speed all = 54.683869185 MB/sec (without seq num) 2024/11/19-15:20:56.990845 139851169007360 [compaction_job.cc:1948] [default] [JOB 139] Generated table #4802: 64714 keys, 66136964 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:56.990898 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000856990873, "cf_name": "default", "job": 139, "event": "table_file_creation", "file_number": 4802, "file_size": 66136964, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025200, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294280, "raw_average_key_size": 20, "raw_value_size": 65814667, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64714, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000856, "oldest_key_time": 0, "file_creation_time": 1732000856, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4802, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:57.006397 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004803.sst) /storage/mytopling/..//tmp/Topling-82038-luABUV 2024/11/19-15:20:57.057865 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004803.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:20:57.061635 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004803.sst): index pass time = 0.00's, 136.122'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63555 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:20:57.078652 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-00 2024/11/19-15:20:57.078726 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00146/att-00/compact.done) = No such file or directory: 7.005 sec on http probe failed: 2, retry 2024/11/19-15:20:57.123290 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004803.sst): first pass time = 0.12's, 563.876'MB/sec 2024/11/19-15:20:57.123350 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004803.sst): old prealloc_size = 73819750, real_size = 64962576 2024/11/19-15:20:57.123408 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004803.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63555 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0646 GB zip-val = 0.0648 GB avg-val =1016.94 avg-zval =1020.19 2024/11/19-15:20:57.123475 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004803.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.066 GB 2024/11/19-15:20:57.182143 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004803.sst): second pass time = 0.00's,233326447.653'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1104.925'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2044.752'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.380'MB/sec write SST data time = 0.02's, 3849.664'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 233326447.653'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63555 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.94 avg-zval = 1020.19 usrkeys = 63555 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.94 avg-zval = 1020.19 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7918 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0648 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index = 7.0429 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 7.264964 GB avg = 0.280 KB (by entry num) total key len = 0.664852 GB avg = 0.026 KB total ukey len = 0.457211 GB avg = 0.018 KB total ukey num = 0.025955143 Billion total entry num = 0.025955143 Billion write speed all = 56.555111902 MB/sec (with seq num) write speed all = 55.074224220 MB/sec (without seq num) 2024/11/19-15:20:57.186112 139851169007360 [compaction_job.cc:1948] [default] [JOB 139] Generated table #4803: 63555 keys, 64948036 bytes (need compaction), temperature: kUnknown 2024/11/19-15:20:57.186168 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000857186143, "cf_name": "default", "job": 139, "event": "table_file_creation", "file_number": 4803, "file_size": 64948036, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64838208, "index_size": 108288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1271100, "raw_average_key_size": 20, "raw_value_size": 64631426, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63555, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000857, "oldest_key_time": 0, "file_creation_time": 1732000857, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4803, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:57.201028 139851169007360 [compaction_job.cc:703] job-00139: subcompact[0], size: 0.131085 G, files: /storage/mytopling/.rocksdb [004802,004803] 2024/11/19-15:20:57.201763 139851169007360 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777214 2024/11/19-15:20:57.201891 139851169007360 (Original Log Time 2024/11/19-15:20:56.792268) EVENT_LOG_v1 {"time_micros": 1732000856792240, "job": 139, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4699, 4700, 4701, 4702], "score": 2.66901, "input_data_size": 134100352, "oldest_snapshot_seqno": -1} 2024/11/19-15:20:57.201898 139851169007360 (Original Log Time 2024/11/19-15:20:57.201480) [compaction_job.cc:2022] [default] [JOB 139] Compacted 4@1 files to L2 => 131085000 bytes 2024/11/19-15:20:57.201901 139851169007360 (Original Log Time 2024/11/19-15:20:57.201730) [compaction_job.cc:1224] [default] compacted to: files[6 39 59 18 0 0 0] max score 2.50 (86 files need compaction), MB/sec: 328.1 rd, 320.7 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128269, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:57.201916 139851169007360 (Original Log Time 2024/11/19-15:20:57.201756) EVENT_LOG_v1 {"time_micros": 1732000857201743, "cf": "default", "job": 139, "event": "compaction_finished", "compaction_time_micros": 408774, "compaction_time_cpu_micros": 406522, "output_level": 2, "num_output_files": 2, "total_output_size": 131085000, "num_input_records": 128269, "num_output_records": 128269, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 39, 59, 18, 0, 0, 0]} 2024/11/19-15:20:57.206058 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004684.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:57.206083 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000857206080, "job": 139, "event": "table_file_deletion", "file_number": 4684} 2024/11/19-15:20:57.209928 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004685.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:20:57.209948 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000857209946, "job": 139, "event": "table_file_deletion", "file_number": 4685} 2024/11/19-15:20:57.210077 139851169007360 [compaction_job.cc:2437] [default] [JOB 150] Compacting 1@2 files to L3, score 1.88, subcompactions 1 : 1 2024/11/19-15:20:57.210085 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 148 Base level 2, inputs: [4755(644KB)] 2024/11/19-15:20:57.421507 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-00 2024/11/19-15:20:57.421612 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00149/att-00/compact.done) = No such file or directory: 2.214 sec on http probe failed: 3, retry 2024/11/19-15:20:57.499607 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-00 2024/11/19-15:20:57.499687 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00147/att-00/compact.done) = No such file or directory: 5.671 sec on http probe failed: 5, giveup 2024/11/19-15:20:57.499709 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00147/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 53.697 sec, real 5.996 (submit 316.745 ms wait 5.671 s), input 53.697 MiB, send shutdown 2024/11/19-15:20:57.508535 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-00 2024/11/19-15:20:57.508784 139851158521600 [dcompact_etcd.cc:1773] job-00147/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:57.510880 139851169007360 [dcompact_etcd.cc:1163] job-00150/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.300 sec, input = 3.296 MiB 2024/11/19-15:20:57.598290 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-00 2024/11/19-15:20:57.598385 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00150/att-00/compact.done) = No such file or directory: 0.082 sec on http probe failed: 1, retry 2024/11/19-15:20:57.671950 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-00 2024/11/19-15:20:57.672036 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00150/att-00/compact.done) = No such file or directory: 0.153 sec on http probe failed: 2, retry 2024/11/19-15:20:57.742773 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-00 2024/11/19-15:20:57.742840 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00150/att-00/compact.done) = No such file or directory: 0.227 sec on http probe failed: 3, retry 2024/11/19-15:20:57.813411 139851158521600 [dcompact_etcd.cc:1163] job-00147/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.305 sec, input = 53.697 MiB 2024/11/19-15:20:57.817347 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-00 2024/11/19-15:20:57.817403 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00150/att-00/compact.done) = No such file or directory: 0.298 sec on http probe failed: 4, retry 2024/11/19-15:20:57.890250 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-00 2024/11/19-15:20:57.890306 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00150/att-00/compact.done) = No such file or directory: 0.373 sec on http probe failed: 5, giveup 2024/11/19-15:20:57.890321 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00150/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.680 (submit 300.359 ms wait 0.373 s), input 3.296 MiB, send shutdown 2024/11/19-15:20:57.898694 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-00 2024/11/19-15:20:57.898898 139851169007360 [dcompact_etcd.cc:1773] job-00150/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:58.104436 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-00 2024/11/19-15:20:58.104515 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00149/att-00/compact.done) = No such file or directory: 2.900 sec on http probe failed: 4, retry 2024/11/19-15:20:58.150970 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-01 2024/11/19-15:20:58.151063 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00142/att-01/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:20:58.188406 139851169007360 [dcompact_etcd.cc:1163] job-00150/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 3.296 MiB 2024/11/19-15:20:58.280888 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-01 2024/11/19-15:20:58.280980 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00150/att-01/compact.done) = No such file or directory: 0.082 sec on http probe failed: 1, retry 2024/11/19-15:20:58.299310 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-01 2024/11/19-15:20:58.299399 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00143/att-01/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:20:58.323891 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-02 2024/11/19-15:20:58.323954 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00141/att-02/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:20:58.355833 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-01 2024/11/19-15:20:58.355888 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00150/att-01/compact.done) = No such file or directory: 0.159 sec on http probe failed: 2, retry 2024/11/19-15:20:58.427031 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-01 2024/11/19-15:20:58.427088 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00150/att-01/compact.done) = No such file or directory: 0.233 sec on http probe failed: 3, retry 2024/11/19-15:20:58.502089 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-01 2024/11/19-15:20:58.502142 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00150/att-01/compact.done) = No such file or directory: 0.305 sec on http probe failed: 4, retry 2024/11/19-15:20:58.573242 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-01 2024/11/19-15:20:58.573316 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00150/att-01/compact.done) = No such file or directory: 0.380 sec on http probe failed: 5, giveup 2024/11/19-15:20:58.573333 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00150/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.674 (submit 289.413 ms wait 0.380 s), input 3.296 MiB, send shutdown 2024/11/19-15:20:58.578343 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-01 2024/11/19-15:20:58.578559 139851169007360 [dcompact_etcd.cc:1773] job-00150/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:58.789415 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-00 2024/11/19-15:20:58.789493 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00149/att-00/compact.done) = No such file or directory: 3.582 sec on http probe failed: 5, giveup 2024/11/19-15:20:58.789511 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00149/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 33.874 sec, real 3.880 (submit 290.033 ms wait 3.582 s), input 33.874 MiB, send shutdown 2024/11/19-15:20:58.793949 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-00 2024/11/19-15:20:58.794188 139851179493120 [dcompact_etcd.cc:1773] job-00149/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:58.843548 139851169007360 [dcompact_etcd.cc:1163] job-00150/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.265 sec, input = 3.296 MiB 2024/11/19-15:20:58.934634 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-02 2024/11/19-15:20:58.934706 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00150/att-02/compact.done) = No such file or directory: 0.082 sec on http probe failed: 1, retry 2024/11/19-15:20:59.005596 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-02 2024/11/19-15:20:59.005677 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00150/att-02/compact.done) = No such file or directory: 0.157 sec on http probe failed: 2, retry 2024/11/19-15:20:59.067950 139851179493120 [dcompact_etcd.cc:1163] job-00149/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.274 sec, input = 33.874 MiB 2024/11/19-15:20:59.076617 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-02 2024/11/19-15:20:59.076685 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00150/att-02/compact.done) = No such file or directory: 0.228 sec on http probe failed: 3, retry 2024/11/19-15:20:59.087312 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-00 2024/11/19-15:20:59.087388 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00146/att-00/compact.done) = No such file or directory: 9.011 sec on http probe failed: 3, retry 2024/11/19-15:20:59.147192 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-02 2024/11/19-15:20:59.147253 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00150/att-02/compact.done) = No such file or directory: 0.299 sec on http probe failed: 4, retry 2024/11/19-15:20:59.165489 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-01 2024/11/19-15:20:59.165552 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00147/att-01/compact.done) = No such file or directory: 1.343 sec on http probe failed: 1, retry 2024/11/19-15:20:59.217940 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-02 2024/11/19-15:20:59.217994 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00150/att-02/compact.done) = No such file or directory: 0.370 sec on http probe failed: 5, giveup 2024/11/19-15:20:59.218010 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00150/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.639 (submit 264.907 ms wait 0.370 s), input 3.296 MiB, send shutdown 2024/11/19-15:20:59.223348 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00150/att-02 2024/11/19-15:20:59.223557 139851169007360 [dcompact_etcd.cc:1773] job-00150/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:20:59.225311 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00150) = success 2024/11/19-15:20:59.225318 139851169007360 [dcompact_etcd.cc:1773] job-00150/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:20:59.225337 139851169007360 (Original Log Time 2024/11/19-15:20:57.210097) EVENT_LOG_v1 {"time_micros": 1732000857210090, "job": 150, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4755], "score": 1.88083, "input_data_size": 660390, "oldest_snapshot_seqno": 130461071} 2024/11/19-15:20:59.225344 139851169007360 [compaction_job.cc:2437] [default] [JOB 150] Compacting 1@2 files to L3, score 1.88, subcompactions 1 : 1 2024/11/19-15:20:59.225351 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 148 Base level 2, inputs: [4755(644KB)] 2024/11/19-15:20:59.225631 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004804.sst) /storage/mytopling/..//tmp/Topling-82038-IWEtWZ 2024/11/19-15:20:59.264450 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004804.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0013 GB, waited 0.000 sec, Key+Value = 0.003 GB 2024/11/19-15:20:59.266189 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004804.sst): first pass time = 0.04's, 85.232'MB/sec 2024/11/19-15:20:59.271900 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004804.sst): index pass time = 0.01's, 232.698'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 144000 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0017 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:20:59.271966 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004804.sst): old prealloc_size = 726429, real_size = 280416 2024/11/19-15:20:59.272041 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004804.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZeroLengthBlobStore usrkeys = 144000 prefix = 4 raw-key = 0.0017 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 0.00 avg-zval = 0.00 2024/11/19-15:20:59.272093 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004804.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.008 sec, Key+Value = 0.003 GB 2024/11/19-15:20:59.284080 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004804.sst): second pass time = 0.00's, 0.000'MB/sec, value only( 0.0% of KV) wait indexing time = 0.01's, remap KeyValue time = 0.01's, 20.349'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3394.600'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 164.177'MB/sec write SST data time = 0.01's, 20.849'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 0.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 144000 avg-key = 24.00 avg-zkey = 1.70 avg-val = 0.00 avg-zval = 0.00 usrkeys = 144000 avg-key = 16.00 avg-zkey = 1.70 avg-val = 0.00 avg-zval = 0.00 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 5 UnZipSize{ index = 0.0023 GB value = 0.0000 GB dict = 0.00 MB all = 0.0035 GB } __ZipSize{ index = 0.0002 GB value = 0.0000 GB dict = 0.00 MB all = 0.0002 GB } UnZip/Zip{ index = 9.4364 value = 0.0000 dict = 0.00 all = 14.0318 } Zip/UnZip{ index = 0.1060 value = inf dict = 0.00 all = 0.0713 } ---------------------------- total value len = 7.264964 GB avg = 0.278 KB (by entry num) total key len = 0.668308 GB avg = 0.026 KB total ukey len = 0.459515 GB avg = 0.018 KB total ukey num = 0.026099143 Billion total entry num = 0.026099143 Billion write speed all = 55.744101603 MB/sec (with seq num) write speed all = 54.276991317 MB/sec (without seq num) 2024/11/19-15:20:59.284401 139851169007360 [compaction_job.cc:1948] [default] [JOB 150] Generated table #4804: 144000 keys, 246297 bytes, temperature: kUnknown 2024/11/19-15:20:59.284449 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000859284424, "cf_name": "default", "job": 150, "event": "table_file_creation", "file_number": 4804, "file_size": 246297, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 80, "index_size": 244160, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3456000, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 144000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZeroLengthBlobStore", "creation_time": 1732000859, "oldest_key_time": 0, "file_creation_time": 1732000859, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4804, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:20:59.284940 139851169007360 [compaction_job.cc:703] job-00150: subcompact[0], size: 0.000246 G, files: /storage/mytopling/.rocksdb [004804] 2024/11/19-15:20:59.285470 139851169007360 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777216 2024/11/19-15:20:59.285702 139851169007360 (Original Log Time 2024/11/19-15:20:59.225380) EVENT_LOG_v1 {"time_micros": 1732000859225364, "job": 150, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4755], "score": 1.88083, "input_data_size": 660390, "oldest_snapshot_seqno": 130461071} 2024/11/19-15:20:59.285709 139851169007360 (Original Log Time 2024/11/19-15:20:59.285212) [compaction_job.cc:2022] [default] [JOB 150] Compacted 1@2 files to L3 => 246297 bytes 2024/11/19-15:20:59.285712 139851169007360 (Original Log Time 2024/11/19-15:20:59.285446) [compaction_job.cc:1224] [default] compacted to: files[6 39 58 19 0 0 0] max score 2.50 (85 files need compaction), MB/sec: 11.1 rd, 4.1 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(0.6, 0.0 +0.0 blob) out(0.2 +0.0 blob), read-write-amplify(1.4) write-amplify(0.4) OK, records in: 144000, records dropped: 0 output_compression: Snappy 2024/11/19-15:20:59.285725 139851169007360 (Original Log Time 2024/11/19-15:20:59.285465) EVENT_LOG_v1 {"time_micros": 1732000859285454, "cf": "default", "job": 150, "event": "compaction_finished", "compaction_time_micros": 59567, "compaction_time_cpu_micros": 53565, "output_level": 3, "num_output_files": 1, "total_output_size": 246297, "num_input_records": 144000, "num_output_records": 144000, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 39, 58, 19, 0, 0, 0]} 2024/11/19-15:20:59.285800 139851169007360 [compaction_job.cc:2437] [default] [JOB 151] Compacting 1@2 files to L3, score 1.88, subcompactions 1 : 1 2024/11/19-15:20:59.285808 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 149 Base level 2, inputs: [4756(11MB)] 2024/11/19-15:20:59.601894 139851169007360 [dcompact_etcd.cc:1163] job-00151/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.316 sec, input = 20.050 MiB 2024/11/19-15:20:59.922073 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-01 2024/11/19-15:20:59.922152 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00149/att-01/compact.done) = No such file or directory: 0.847 sec on http probe failed: 1, retry 2024/11/19-15:21:00.111494 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-00 2024/11/19-15:21:00.111593 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00151/att-00/compact.done) = No such file or directory: 0.501 sec on http probe failed: 1, retry 2024/11/19-15:21:00.159192 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-01 2024/11/19-15:21:00.159258 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00142/att-01/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:21:00.249548 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-01 2024/11/19-15:21:00.249606 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00147/att-01/compact.done) = No such file or directory: 2.426 sec on http probe failed: 2, retry 2024/11/19-15:21:00.305178 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-01 2024/11/19-15:21:00.305251 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00143/att-01/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:21:00.328452 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-02 2024/11/19-15:21:00.328518 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00141/att-02/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:21:00.522138 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-00 2024/11/19-15:21:00.522214 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00151/att-00/compact.done) = No such file or directory: 0.911 sec on http probe failed: 2, retry 2024/11/19-15:21:00.608865 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-01 2024/11/19-15:21:00.608930 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00149/att-01/compact.done) = No such file or directory: 1.532 sec on http probe failed: 2, retry 2024/11/19-15:21:00.928047 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-00 2024/11/19-15:21:00.928134 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00151/att-00/compact.done) = No such file or directory: 1.321 sec on http probe failed: 3, retry 2024/11/19-15:21:01.093056 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-00 2024/11/19-15:21:01.093152 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00146/att-00/compact.done) = No such file or directory: 11.019 sec on http probe failed: 4, retry 2024/11/19-15:21:01.294830 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-01 2024/11/19-15:21:01.294904 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00149/att-01/compact.done) = No such file or directory: 2.219 sec on http probe failed: 3, retry 2024/11/19-15:21:01.332309 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-01 2024/11/19-15:21:01.332375 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00147/att-01/compact.done) = No such file or directory: 3.510 sec on http probe failed: 3, retry 2024/11/19-15:21:01.337317 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-00 2024/11/19-15:21:01.337376 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00151/att-00/compact.done) = No such file or directory: 1.727 sec on http probe failed: 4, retry 2024/11/19-15:21:01.744720 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-00 2024/11/19-15:21:01.744796 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00151/att-00/compact.done) = No such file or directory: 2.137 sec on http probe failed: 5, giveup 2024/11/19-15:21:01.744814 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00151/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 20.050 sec, real 2.459 (submit 315.636 ms wait 2.137 s), input 20.050 MiB, send shutdown 2024/11/19-15:21:01.752925 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-00 2024/11/19-15:21:01.753133 139851169007360 [dcompact_etcd.cc:1773] job-00151/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:01.981025 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-01 2024/11/19-15:21:01.981158 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00149/att-01/compact.done) = No such file or directory: 2.905 sec on http probe failed: 4, retry 2024/11/19-15:21:02.043423 139851169007360 [dcompact_etcd.cc:1163] job-00151/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.290 sec, input = 20.050 MiB 2024/11/19-15:21:02.167221 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-01 2024/11/19-15:21:02.167295 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00142/att-01/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:21:02.167314 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00142/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.813 sec, real 11.512 (submit 299.969 ms wait 11.204 s), input 126.813 MiB, send shutdown 2024/11/19-15:21:02.176482 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-01 2024/11/19-15:21:02.176705 139851127064320 [dcompact_etcd.cc:1773] job-00142/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:02.313405 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-01 2024/11/19-15:21:02.313474 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00143/att-01/compact.done) = No such file or directory: 7.185 sec on http probe failed: 3, retry 2024/11/19-15:21:02.333305 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-02 2024/11/19-15:21:02.333362 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00141/att-02/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:21:02.411377 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-01 2024/11/19-15:21:02.411434 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00147/att-01/compact.done) = No such file or directory: 4.593 sec on http probe failed: 4, retry 2024/11/19-15:21:02.498620 139851127064320 [dcompact_etcd.cc:1163] job-00142/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.322 sec, input = 126.813 MiB 2024/11/19-15:21:02.554439 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-01 2024/11/19-15:21:02.554502 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00151/att-01/compact.done) = No such file or directory: 0.501 sec on http probe failed: 1, retry 2024/11/19-15:21:02.668179 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-01 2024/11/19-15:21:02.668257 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00149/att-01/compact.done) = No such file or directory: 3.591 sec on http probe failed: 5, giveup 2024/11/19-15:21:02.668275 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00149/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 33.874 sec, real 3.874 (submit 273.670 ms wait 3.591 s), input 33.874 MiB, send shutdown 2024/11/19-15:21:02.676064 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-01 2024/11/19-15:21:02.676283 139851179493120 [dcompact_etcd.cc:1773] job-00149/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:02.933665 139851179493120 [dcompact_etcd.cc:1163] job-00149/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.257 sec, input = 33.874 MiB 2024/11/19-15:21:02.963188 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-01 2024/11/19-15:21:02.963246 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00151/att-01/compact.done) = No such file or directory: 0.912 sec on http probe failed: 2, retry 2024/11/19-15:21:03.102592 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-00 2024/11/19-15:21:03.102699 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00146/att-00/compact.done) = No such file or directory: 13.025 sec on http probe failed: 5, giveup 2024/11/19-15:21:03.102718 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00146/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 254.479 sec, real 13.311 (submit 276.296 ms wait 13.025 s), input 254.479 MiB, send shutdown 2024/11/19-15:21:03.111762 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-00 2024/11/19-15:21:03.112013 139851137550080 [dcompact_etcd.cc:1773] job-00146/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:03.378557 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-01 2024/11/19-15:21:03.378631 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00151/att-01/compact.done) = No such file or directory: 1.321 sec on http probe failed: 3, retry 2024/11/19-15:21:03.382582 139851137550080 [dcompact_etcd.cc:1163] job-00146/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.270 sec, input = 254.479 MiB 2024/11/19-15:21:03.494287 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-01 2024/11/19-15:21:03.494384 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00147/att-01/compact.done) = No such file or directory: 5.672 sec on http probe failed: 5, giveup 2024/11/19-15:21:03.494408 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00147/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 53.697 sec, real 5.986 (submit 304.525 ms wait 5.672 s), input 53.697 MiB, send shutdown 2024/11/19-15:21:03.503692 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-01 2024/11/19-15:21:03.503956 139851158521600 [dcompact_etcd.cc:1773] job-00147/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:03.787502 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-01 2024/11/19-15:21:03.787588 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00151/att-01/compact.done) = No such file or directory: 1.736 sec on http probe failed: 4, retry 2024/11/19-15:21:03.788684 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-02 2024/11/19-15:21:03.788760 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00149/att-02/compact.done) = No such file or directory: 0.847 sec on http probe failed: 1, retry 2024/11/19-15:21:03.791859 139851158521600 [dcompact_etcd.cc:1163] job-00147/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.288 sec, input = 53.697 MiB 2024/11/19-15:21:04.195725 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-01 2024/11/19-15:21:04.195800 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00151/att-01/compact.done) = No such file or directory: 2.145 sec on http probe failed: 5, giveup 2024/11/19-15:21:04.195818 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00151/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 20.050 sec, real 2.443 (submit 290.188 ms wait 2.145 s), input 20.050 MiB, send shutdown 2024/11/19-15:21:04.200379 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-01 2024/11/19-15:21:04.200583 139851169007360 [dcompact_etcd.cc:1773] job-00151/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:04.322538 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-01 2024/11/19-15:21:04.322643 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00143/att-01/compact.done) = No such file or directory: 9.193 sec on http probe failed: 4, retry 2024/11/19-15:21:04.338483 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-02 2024/11/19-15:21:04.338570 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00141/att-02/compact.done) = No such file or directory: 11.201 sec on http probe failed: 5, giveup 2024/11/19-15:21:04.338594 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00141/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.972 sec, real 11.476 (submit 270.415 ms wait 11.201 s), input 126.972 MiB, send shutdown 2024/11/19-15:21:04.346873 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00141/att-02 2024/11/19-15:21:04.347151 139850978166528 [dcompact_etcd.cc:1773] job-00141/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:04.349081 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00141) = success 2024/11/19-15:21:04.349095 139850978166528 [dcompact_etcd.cc:1773] job-00141/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:21:04.349135 139850978166528 (Original Log Time 2024/11/19-15:20:29.857215) EVENT_LOG_v1 {"time_micros": 1732000829857209, "job": 141, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4703, 4704, 4705, 4706], "score": 2.41923, "input_data_size": 134098368, "oldest_snapshot_seqno": 128915425} 2024/11/19-15:21:04.349146 139850978166528 [compaction_job.cc:2437] [default] [JOB 141] Compacting 4@1 files to L2, score 2.42, subcompactions 1 : 1 2024/11/19-15:21:04.349156 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 139 Base level 1, inputs: [4703(31MB) 4704(31MB) 4705(31MB) 4706(31MB)] 2024/11/19-15:21:04.349529 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004805.sst) /storage/mytopling/..//tmp/Topling-82038-JU2kt3 2024/11/19-15:21:04.421004 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004805.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:21:04.421146 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004805.sst): index pass time = 0.00's,4736.891'MB/sec index type = UintIndex_AllOne usrkeys = 64716 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:21:04.473325 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-02 2024/11/19-15:21:04.473415 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00149/att-02/compact.done) = No such file or directory: 1.533 sec on http probe failed: 2, retry 2024/11/19-15:21:04.488530 139851169007360 [dcompact_etcd.cc:1163] job-00151/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.288 sec, input = 20.050 MiB 2024/11/19-15:21:04.489462 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004805.sst): first pass time = 0.14's, 479.643'MB/sec 2024/11/19-15:21:04.489507 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004805.sst): old prealloc_size = 73819750, real_size = 66041832 2024/11/19-15:21:04.489530 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004805.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64716 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.98 avg-zval =1020.23 2024/11/19-15:21:04.557437 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004805.sst): second pass time = 0.00's,150605901.602'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 972.261'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.830'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.313'MB/sec write SST data time = 0.02's, 4058.586'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 150605901.602'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64716 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.98 avg-zval = 1020.23 usrkeys = 64716 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.98 avg-zval = 1020.23 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.7500 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 7.330779 GB avg = 0.280 KB (by entry num) total key len = 0.669603 GB avg = 0.026 KB total ukey len = 0.460292 GB avg = 0.018 KB total ukey num = 0.026163859 Billion total entry num = 0.026163859 Billion write speed all = 54.207073207 MB/sec (with seq num) write speed all = 52.788874701 MB/sec (without seq num) 2024/11/19-15:21:04.615033 139850978166528 [compaction_job.cc:1948] [default] [JOB 141] Generated table #4805: 64716 keys, 66026634 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:04.615088 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000864615064, "cf_name": "default", "job": 141, "event": "table_file_creation", "file_number": 4805, "file_size": 66026634, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025328, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294320, "raw_average_key_size": 20, "raw_value_size": 65814779, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64716, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000864, "oldest_key_time": 0, "file_creation_time": 1732000864, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4805, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:04.631219 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004806.sst) /storage/mytopling/..//tmp/Topling-82038-AwY5cv 2024/11/19-15:21:04.684880 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004806.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:21:04.688148 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004806.sst): index pass time = 0.00's, 157.235'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63675 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:21:04.750893 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004806.sst): first pass time = 0.12's, 551.853'MB/sec 2024/11/19-15:21:04.750953 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004806.sst): old prealloc_size = 73819750, real_size = 65088736 2024/11/19-15:21:04.751010 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004806.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63675 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0648 GB zip-val = 0.0650 GB avg-val =1016.99 avg-zval =1020.25 2024/11/19-15:21:04.751056 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004806.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.066 GB 2024/11/19-15:21:04.812474 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004806.sst): second pass time = 0.00's,211624254.902'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1058.255'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2049.693'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.369'MB/sec write SST data time = 0.02's, 3561.744'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 211624254.902'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63675 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.25 usrkeys = 63675 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7935 UnZipSize{ index = 0.0008 GB value = 0.0648 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0650 GB dict = 0.00 MB all = 0.0651 GB } UnZip/Zip{ index = 7.0437 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 7.395536 GB avg = 0.282 KB (by entry num) total key len = 0.670876 GB avg = 0.026 KB total ukey len = 0.461056 GB avg = 0.018 KB total ukey num = 0.026227534 Billion total entry num = 0.026227534 Billion write speed all = 54.560189961 MB/sec (with seq num) write speed all = 53.140992281 MB/sec (without seq num) 2024/11/19-15:21:04.911267 139850978166528 [compaction_job.cc:1948] [default] [JOB 141] Generated table #4806: 63675 keys, 65074244 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:04.911327 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000864911302, "cf_name": "default", "job": 141, "event": "table_file_creation", "file_number": 4806, "file_size": 65074244, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64964176, "index_size": 108480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1273500, "raw_average_key_size": 20, "raw_value_size": 64757022, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63675, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000864, "oldest_key_time": 0, "file_creation_time": 1732000864, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4806, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:04.926003 139850978166528 [compaction_job.cc:703] job-00141: subcompact[0], size: 0.131101 G, files: /storage/mytopling/.rocksdb [004805,004806] 2024/11/19-15:21:04.926684 139850978166528 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777216 2024/11/19-15:21:04.926829 139850978166528 (Original Log Time 2024/11/19-15:21:04.349193) EVENT_LOG_v1 {"time_micros": 1732000864349173, "job": 141, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4703, 4704, 4705, 4706], "score": 2.41923, "input_data_size": 134098368, "oldest_snapshot_seqno": 128915425} 2024/11/19-15:21:04.926839 139850978166528 (Original Log Time 2024/11/19-15:21:04.926407) [compaction_job.cc:2022] [default] [JOB 141] Compacted 4@1 files to L2 => 131100878 bytes 2024/11/19-15:21:04.926843 139850978166528 (Original Log Time 2024/11/19-15:21:04.926652) [compaction_job.cc:1224] [default] compacted to: files[6 35 60 19 0 0 0] max score 2.79 (86 files need compaction), MB/sec: 232.5 rd, 227.3 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128391, records dropped: 0 output_compression: Snappy 2024/11/19-15:21:04.926867 139850978166528 (Original Log Time 2024/11/19-15:21:04.926677) EVENT_LOG_v1 {"time_micros": 1732000864926664, "cf": "default", "job": 141, "event": "compaction_finished", "compaction_time_micros": 576824, "compaction_time_cpu_micros": 423485, "output_level": 2, "num_output_files": 2, "total_output_size": 131100878, "num_input_records": 128391, "num_output_records": 128391, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 35, 60, 19, 0, 0, 0]} 2024/11/19-15:21:04.930879 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004676.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:04.930903 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000864930901, "job": 141, "event": "table_file_deletion", "file_number": 4676} 2024/11/19-15:21:04.934766 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004677.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:04.934786 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000864934784, "job": 141, "event": "table_file_deletion", "file_number": 4677} 2024/11/19-15:21:04.938677 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004678.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:04.938697 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000864938695, "job": 141, "event": "table_file_deletion", "file_number": 4678} 2024/11/19-15:21:04.942731 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004679.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:04.942751 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000864942749, "job": 141, "event": "table_file_deletion", "file_number": 4679} 2024/11/19-15:21:04.942877 139850978166528 [compaction_job.cc:2437] [default] [JOB 152] Compacting 1@2 files to L3, score 1.95, subcompactions 1 : 1 2024/11/19-15:21:04.942886 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 150 Base level 2, inputs: [4599(52MB)] 2024/11/19-15:21:04.998848 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-02 2024/11/19-15:21:04.998918 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00151/att-02/compact.done) = No such file or directory: 0.501 sec on http probe failed: 1, retry 2024/11/19-15:21:05.143984 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-02 2024/11/19-15:21:05.144063 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00147/att-02/compact.done) = No such file or directory: 1.342 sec on http probe failed: 1, retry 2024/11/19-15:21:05.155930 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-02 2024/11/19-15:21:05.155976 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00149/att-02/compact.done) = No such file or directory: 2.217 sec on http probe failed: 3, retry 2024/11/19-15:21:05.236595 139850978166528 [dcompact_etcd.cc:1163] job-00152/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.293 sec, input = 63.819 MiB 2024/11/19-15:21:05.404788 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-02 2024/11/19-15:21:05.404861 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00151/att-02/compact.done) = No such file or directory: 0.911 sec on http probe failed: 2, retry 2024/11/19-15:21:05.675912 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-02 2024/11/19-15:21:05.676007 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00142/att-02/compact.done) = No such file or directory: 3.170 sec on http probe failed: 1, retry 2024/11/19-15:21:05.814685 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-02 2024/11/19-15:21:05.814749 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00151/att-02/compact.done) = No such file or directory: 1.317 sec on http probe failed: 3, retry 2024/11/19-15:21:05.839156 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-02 2024/11/19-15:21:05.839213 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00149/att-02/compact.done) = No such file or directory: 2.900 sec on http probe failed: 4, retry 2024/11/19-15:21:06.223996 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-02 2024/11/19-15:21:06.224089 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00151/att-02/compact.done) = No such file or directory: 1.727 sec on http probe failed: 4, retry 2024/11/19-15:21:06.227048 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-02 2024/11/19-15:21:06.227116 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00147/att-02/compact.done) = No such file or directory: 2.426 sec on http probe failed: 2, retry 2024/11/19-15:21:06.331008 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-01 2024/11/19-15:21:06.331076 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00143/att-01/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:21:06.331094 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00143/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.830 sec, real 11.465 (submit 254.562 ms wait 11.202 s), input 126.830 MiB, send shutdown 2024/11/19-15:21:06.340130 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-01 2024/11/19-15:21:06.340351 139851148035840 [dcompact_etcd.cc:1773] job-00143/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:06.522863 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-02 2024/11/19-15:21:06.522947 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00149/att-02/compact.done) = No such file or directory: 3.583 sec on http probe failed: 5, giveup 2024/11/19-15:21:06.522965 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00149/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 33.874 sec, real 3.847 (submit 257.295 ms wait 3.583 s), input 33.874 MiB, send shutdown 2024/11/19-15:21:06.527536 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00149/att-02 2024/11/19-15:21:06.527745 139851179493120 [dcompact_etcd.cc:1773] job-00149/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:06.527970 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00149) = success 2024/11/19-15:21:06.527976 139851179493120 [dcompact_etcd.cc:1773] job-00149/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:06.527993 139851179493120 (Original Log Time 2024/11/19-15:20:54.909116) EVENT_LOG_v1 {"time_micros": 1732000854909109, "job": 149, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4757], "score": 1.82029, "input_data_size": 27421320, "oldest_snapshot_seqno": 130221952} 2024/11/19-15:21:06.528000 139851179493120 [compaction_job.cc:2437] [default] [JOB 149] Compacting 1@2 files to L3, score 1.82, subcompactions 1 : 1 2024/11/19-15:21:06.528007 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 147 Base level 2, inputs: [4757(26MB)] 2024/11/19-15:21:06.528250 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004807.sst) /storage/mytopling/..//tmp/Topling-82038-21qhE0 2024/11/19-15:21:06.602550 139851148035840 [dcompact_etcd.cc:1163] job-00143/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.262 sec, input = 126.830 MiB 2024/11/19-15:21:06.634896 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-02 2024/11/19-15:21:06.634959 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00151/att-02/compact.done) = No such file or directory: 2.137 sec on http probe failed: 5, giveup 2024/11/19-15:21:06.634975 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00151/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 20.050 sec, real 2.434 (submit 287.858 ms wait 2.137 s), input 20.050 MiB, send shutdown 2024/11/19-15:21:06.642181 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00151/att-02 2024/11/19-15:21:06.642387 139851169007360 [dcompact_etcd.cc:1773] job-00151/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:06.644111 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00151) = success 2024/11/19-15:21:06.644119 139851169007360 [dcompact_etcd.cc:1773] job-00151/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:21:06.644136 139851169007360 (Original Log Time 2024/11/19-15:20:59.285820) EVENT_LOG_v1 {"time_micros": 1732000859285814, "job": 151, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4756], "score": 1.87868, "input_data_size": 12168541, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:06.644143 139851169007360 [compaction_job.cc:2437] [default] [JOB 151] Compacting 1@2 files to L3, score 1.88, subcompactions 1 : 1 2024/11/19-15:21:06.644150 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 149 Base level 2, inputs: [4756(11MB)] 2024/11/19-15:21:06.644395 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004808.sst) /storage/mytopling/..//tmp/Topling-82038-cY29MK 2024/11/19-15:21:06.657141 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004807.sst): first pass time = 0.13's, 275.605'MB/sec 2024/11/19-15:21:06.657145 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004807.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.036 GB 2024/11/19-15:21:06.657171 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004807.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, dictZip workingMem = 0.0016 GB, waited 0.000 sec, Key+Value = 0.036 GB 2024/11/19-15:21:06.657228 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004807.sst): index pass time = 0.00's,64341.007'MB/sec index type = UintIndex_AllOne usrkeys = 480000 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0038 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:21:06.795181 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004808.sst: sumWaitingMem = 0.004 GB, sumWorkingMem = 0.002 GB, nltTrie workingMem = 0.0043 GB, waited 0.000 sec, Key+Value = 0.021 GB 2024/11/19-15:21:06.851508 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004808.sst): first pass time = 0.21's, 101.516'MB/sec 2024/11/19-15:21:06.853091 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-00 2024/11/19-15:21:06.853200 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00152/att-00/compact.done) = No such file or directory: 1.596 sec on http probe failed: 1, retry 2024/11/19-15:21:06.872228 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004808.sst): index pass time = 0.08's, 74.802'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 480000 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0058 GB zip-key = 0.0008 GB avg-key = 12.00 avg-zkey = 1.68 2024/11/19-15:21:06.872326 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004808.sst): old prealloc_size = 13385395, real_size = 10469584 2024/11/19-15:21:06.872552 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004808.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 480000 prefix = 4 raw-key = 0.0058 GB zip-key = 0.0008 GB avg-key = 12.00 avg-zkey = 1.68 raw-val = 0.0095 GB zip-val = 0.0095 GB avg-val = 19.80 avg-zval = 19.88 2024/11/19-15:21:06.872613 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004808.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.002 GB, reorder workingMem = 0.0011 GB, waited 0.077 sec, Key+Value = 0.021 GB 2024/11/19-15:21:07.029243 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004808.sst): second pass time = 0.00's,15206400.000'MB/sec, value only(45.2% of KV) wait indexing time = 0.02's, remap KeyValue time = 0.16's, 65.986'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3790.204'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 1.386'MB/sec write SST data time = 0.07's, 147.605'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.010'GB zip my value throughput = 15206400.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 480000 avg-key = 24.00 avg-zkey = 1.68 avg-val = 19.80 avg-zval = 19.88 usrkeys = 480000 avg-key = 16.00 avg-zkey = 1.68 avg-val = 19.80 avg-zval = 19.88 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 1166 UnZipSize{ index = 0.0077 GB value = 0.0095 GB dict = 0.00 MB all = 0.0210 GB } __ZipSize{ index = 0.0008 GB value = 0.0095 GB dict = 0.00 MB all = 0.0104 GB } UnZip/Zip{ index = 9.5042 value = 0.9961 dict = 0.00 all = 2.0304 } Zip/UnZip{ index = 0.1052 value = 1.0039 dict = 0.00 all = 0.4925 } ---------------------------- total value len = 7.405040 GB avg = 0.277 KB (by entry num) total key len = 0.682396 GB avg = 0.026 KB total ukey len = 0.468736 GB avg = 0.018 KB total ukey num = 0.026707534 Billion total entry num = 0.026707534 Billion write speed all = 53.894299580 MB/sec (with seq num) write speed all = 52.470477493 MB/sec (without seq num) 2024/11/19-15:21:07.030399 139851169007360 [compaction_job.cc:1948] [default] [JOB 151] Generated table #4808: 480000 keys, 10354513 bytes, temperature: kUnknown 2024/11/19-15:21:07.030450 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000867030426, "cf_name": "default", "job": 151, "event": "table_file_creation", "file_number": 4808, "file_size": 10354513, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 9541312, "index_size": 808064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 11520000, "raw_average_key_size": 24, "raw_value_size": 9504000, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 480000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732000866, "oldest_key_time": 0, "file_creation_time": 1732000866, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4808, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:07.034021 139851169007360 [compaction_job.cc:703] job-00151: subcompact[0], size: 0.010355 G, files: /storage/mytopling/.rocksdb [004808] 2024/11/19-15:21:07.034738 139851169007360 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777216 2024/11/19-15:21:07.034853 139851169007360 (Original Log Time 2024/11/19-15:21:06.644179) EVENT_LOG_v1 {"time_micros": 1732000866644162, "job": 151, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4756], "score": 1.87868, "input_data_size": 12168541, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:07.034859 139851169007360 (Original Log Time 2024/11/19-15:21:07.034490) [compaction_job.cc:2022] [default] [JOB 151] Compacted 1@2 files to L3 => 10354513 bytes 2024/11/19-15:21:07.034863 139851169007360 (Original Log Time 2024/11/19-15:21:07.034710) [compaction_job.cc:1224] [default] compacted to: files[6 35 59 20 0 0 0] max score 2.79 (85 files need compaction), MB/sec: 31.2 rd, 26.6 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(11.6, 0.0 +0.0 blob) out(9.9 +0.0 blob), read-write-amplify(1.9) write-amplify(0.9) OK, records in: 480000, records dropped: 0 output_compression: Snappy 2024/11/19-15:21:07.034876 139851169007360 (Original Log Time 2024/11/19-15:21:07.034731) EVENT_LOG_v1 {"time_micros": 1732000867034720, "cf": "default", "job": 151, "event": "compaction_finished", "compaction_time_micros": 389855, "compaction_time_cpu_micros": 318283, "output_level": 3, "num_output_files": 1, "total_output_size": 10354513, "num_input_records": 480000, "num_output_records": 480000, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 35, 59, 20, 0, 0, 0]} 2024/11/19-15:21:07.034963 139851169007360 [compaction_job.cc:2437] [default] [JOB 153] Compacting 4@2 files to L3, score 1.91, subcompactions 1 : 1 2024/11/19-15:21:07.034972 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 151 Base level 2, inputs: [4600(52MB) 4601(30MB) 4770(52MB) 4771(52MB)] 2024/11/19-15:21:07.154805 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004807.sst): old prealloc_size = 30163452, real_size = 15953544 2024/11/19-15:21:07.154843 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004807.sst): index type = UintIndex_AllOne , store type = DictZipBlobStore usrkeys = 480000 prefix = 4 raw-key = 0.0038 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0259 GB zip-val = 0.0158 GB avg-val = 54.00 avg-zval = 32.99 2024/11/19-15:21:07.191658 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004807.sst): second pass time = 0.50's, 52.088'MB/sec, value only(73.0% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.04's, 436.933'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.175'MB/sec (index lex order gen) rebuild zvType time = 0.01's, 14.719'MB/sec write SST data time = 0.03's, 561.521'MB/sec dict compress time = 0.00's, 1203.726'MB/sec z-dict build time = 0.00's, sample length = 0.260'MB, throughput = 3912.503'MB/sec zip my value time = 0.50's, unzip length = 0.026'GB zip my value throughput = 52.088'MB/sec zip pipeline throughput = 52.088'MB/sec entries = 480000 avg-key = 20.00 avg-zkey = 0.00 avg-val = 54.00 avg-zval = 32.99 usrkeys = 480000 avg-key = 12.00 avg-zkey = 0.00 avg-val = 54.00 avg-zval = 32.99 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 4 UnZipSize{ index = 0.0058 GB value = 0.0259 GB dict = 0.26 MB all = 0.0355 GB } __ZipSize{ index = 0.0000 GB value = 0.0158 GB dict = 0.26 MB all = 0.0161 GB } UnZip/Zip{ index =30000.0000 value = 1.6371 dict = 1.00 all = 2.2070 } Zip/UnZip{ index = 0.0000 value = 0.6109 dict = 1.00 all = 0.4531 } ---------------------------- total value len = 7.430960 GB avg = 0.273 KB (by entry num) total key len = 0.691996 GB avg = 0.025 KB total ukey len = 0.474496 GB avg = 0.017 KB total ukey num = 0.027187534 Billion total entry num = 0.027187534 Billion write speed all = 54.072477845 MB/sec (with seq num) write speed all = 52.624633211 MB/sec (without seq num) 2024/11/19-15:21:07.192308 139851179493120 [compaction_job.cc:1948] [default] [JOB 149] Generated table #4807: 480000 keys, 16094123 bytes, temperature: kUnknown 2024/11/19-15:21:07.192355 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000867192332, "cf_name": "default", "job": 149, "event": "table_file_creation", "file_number": 4807, "file_size": 16094123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 15833168, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 9600000, "raw_average_key_size": 20, "raw_value_size": 25919770, "raw_average_value_size": 53, "num_data_blocks": 1, "num_entries": 480000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-DictZipBlobStore", "creation_time": 1732000866, "oldest_key_time": 0, "file_creation_time": 1732000866, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4807, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:07.198585 139851179493120 [compaction_job.cc:703] job-00149: subcompact[0], size: 0.016094 G, files: /storage/mytopling/.rocksdb [004807] 2024/11/19-15:21:07.199180 139851179493120 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777216 2024/11/19-15:21:07.199295 139851179493120 (Original Log Time 2024/11/19-15:21:06.528037) EVENT_LOG_v1 {"time_micros": 1732000866528020, "job": 149, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4757], "score": 1.82029, "input_data_size": 27421320, "oldest_snapshot_seqno": 130221952} 2024/11/19-15:21:07.199304 139851179493120 (Original Log Time 2024/11/19-15:21:07.198906) [compaction_job.cc:2022] [default] [JOB 149] Compacted 1@2 files to L3 => 16094123 bytes 2024/11/19-15:21:07.199308 139851179493120 (Original Log Time 2024/11/19-15:21:07.199151) [compaction_job.cc:1224] [default] compacted to: files[6 35 58 21 0 0 0] max score 2.79 (81 files need compaction), MB/sec: 40.9 rd, 24.0 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(26.2, 0.0 +0.0 blob) out(15.3 +0.0 blob), read-write-amplify(1.6) write-amplify(0.6) OK, records in: 480000, records dropped: 0 output_compression: Snappy 2024/11/19-15:21:07.199328 139851179493120 (Original Log Time 2024/11/19-15:21:07.199174) EVENT_LOG_v1 {"time_micros": 1732000867199163, "cf": "default", "job": 149, "event": "compaction_finished", "compaction_time_micros": 670559, "compaction_time_cpu_micros": 480289, "output_level": 3, "num_output_files": 1, "total_output_size": 16094123, "num_input_records": 480000, "num_output_records": 480000, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 35, 58, 21, 0, 0, 0]} 2024/11/19-15:21:07.199420 139851179493120 [compaction_job.cc:2437] [default] [JOB 154] Compacting 4@2 files to L3, score 1.76, subcompactions 1 : 1 2024/11/19-15:21:07.199430 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 152 Base level 2, inputs: [4773(52MB) 4775(52MB) 4772(52MB) 4774(34MB)] 2024/11/19-15:21:07.310317 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-02 2024/11/19-15:21:07.310393 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00147/att-02/compact.done) = No such file or directory: 3.509 sec on http probe failed: 3, retry 2024/11/19-15:21:07.341962 139851169007360 [dcompact_etcd.cc:1163] job-00153/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.307 sec, input = 229.044 MiB 2024/11/19-15:21:07.496068 139851179493120 [dcompact_etcd.cc:1163] job-00154/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.296 sec, input = 233.065 MiB 2024/11/19-15:21:07.683982 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-02 2024/11/19-15:21:07.684052 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00142/att-02/compact.done) = No such file or directory: 5.177 sec on http probe failed: 2, retry 2024/11/19-15:21:08.140060 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-00 2024/11/19-15:21:08.140164 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00152/att-00/compact.done) = No such file or directory: 2.893 sec on http probe failed: 2, retry 2024/11/19-15:21:08.391735 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-01 2024/11/19-15:21:08.391873 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00146/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:21:08.393375 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-02 2024/11/19-15:21:08.393458 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00147/att-02/compact.done) = No such file or directory: 4.593 sec on http probe failed: 4, retry 2024/11/19-15:21:08.496940 139852362282752 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16273899 2024/11/19-15:21:09.426758 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-00 2024/11/19-15:21:09.426833 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00152/att-00/compact.done) = No such file or directory: 4.180 sec on http probe failed: 3, retry 2024/11/19-15:21:09.472571 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-02 2024/11/19-15:21:09.472630 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00147/att-02/compact.done) = No such file or directory: 5.676 sec on http probe failed: 5, giveup 2024/11/19-15:21:09.472648 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00147/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 53.697 sec, real 5.969 (submit 287.784 ms wait 5.676 s), input 53.697 MiB, send shutdown 2024/11/19-15:21:09.476966 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00147/att-02 2024/11/19-15:21:09.477176 139851158521600 [dcompact_etcd.cc:1773] job-00147/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:09.477402 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00147) = success 2024/11/19-15:21:09.477407 139851158521600 [dcompact_etcd.cc:1773] job-00147/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:09.477425 139851158521600 (Original Log Time 2024/11/19-15:20:51.503088) EVENT_LOG_v1 {"time_micros": 1732000851503081, "job": 147, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4761], "score": 1.77263, "input_data_size": 54703820, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:09.477432 139851158521600 [compaction_job.cc:2437] [default] [JOB 147] Compacting 1@2 files to L3, score 1.77, subcompactions 1 : 1 2024/11/19-15:21:09.477439 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 145 Base level 2, inputs: [4761(52MB)] 2024/11/19-15:21:09.477671 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004809.sst) /storage/mytopling/..//tmp/Topling-82038-lDT1LD 2024/11/19-15:21:09.532903 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004809.sst): first pass time = 0.06's,1019.629'MB/sec 2024/11/19-15:21:09.532905 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004809.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0008 GB, waited 0.000 sec, Key+Value = 0.056 GB 2024/11/19-15:21:09.532953 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004809.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0034 GB, waited 0.000 sec, Key+Value = 0.056 GB 2024/11/19-15:21:09.537927 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004809.sst): index pass time = 0.00's, 202.925'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 84154 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0001 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:21:09.693183 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-02 2024/11/19-15:21:09.693349 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00142/att-02/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:21:09.789466 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-02 2024/11/19-15:21:09.789566 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00143/att-02/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:21:10.408252 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-01 2024/11/19-15:21:10.408360 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00146/att-01/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:21:10.474075 139851158521600 [top_zip_table_builder.cc:2752] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004809.sst): part-0: reject ZipStore ratio = 0.804 2024/11/19-15:21:10.570469 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004809.sst): old prealloc_size = 60174202, real_size = 54538640 2024/11/19-15:21:10.570556 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004809.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 84154 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0001 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0543 GB zip-val = 0.0544 GB avg-val = 645.07 avg-zval = 646.13 2024/11/19-15:21:10.570609 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004809.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 1.038 sec, Key+Value = 0.056 GB 2024/11/19-15:21:10.663374 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004809.sst): second pass time = 0.94's, 57.682'MB/sec, value only(96.4% of KV) wait indexing time = 0.10's, remap KeyValue time = 0.09's, 586.876'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2156.859'MB/sec (index lex order gen) rebuild zvType time = 0.07's, 0.286'MB/sec write SST data time = 0.02's, 2827.993'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = 0.000'MB/sec zip my value time = 0.94's, unzip length = 0.054'GB zip my value throughput = 57.682'MB/sec zip pipeline throughput = 57.682'MB/sec entries = 84154 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.07 avg-zval = 646.13 usrkeys = 84154 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.07 avg-zval = 646.13 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6632 UnZipSize{ index = 0.0013 GB value = 0.0543 GB dict = 0.00 MB all = 0.0563 GB } __ZipSize{ index = 0.0001 GB value = 0.0544 GB dict = 0.00 MB all = 0.0545 GB } UnZip/Zip{ index = 9.4174 value = 0.9984 dict = 0.00 all = 1.0327 } Zip/UnZip{ index = 0.1062 value = 1.0016 dict = 0.00 all = 0.9683 } ---------------------------- total value len = 7.485245 GB avg = 0.274 KB (by entry num) total key len = 0.694016 GB avg = 0.025 KB total ukey len = 0.475842 GB avg = 0.017 KB total ukey num = 0.027271688 Billion total entry num = 0.027271688 Billion write speed all = 53.217414657 MB/sec (with seq num) write speed all = 51.797894040 MB/sec (without seq num) 2024/11/19-15:21:10.666851 139851158521600 [compaction_job.cc:1948] [default] [JOB 147] Generated table #4809: 84154 keys, 54519376 bytes, temperature: kUnknown 2024/11/19-15:21:10.666905 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000870666881, "cf_name": "default", "job": 147, "event": "table_file_creation", "file_number": 4809, "file_size": 54519376, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54374464, "index_size": 142976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2019696, "raw_average_key_size": 24, "raw_value_size": 54285186, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 84154, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732000869, "oldest_key_time": 0, "file_creation_time": 1732000869, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4809, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:10.684499 139851158521600 [compaction_job.cc:703] job-00147: subcompact[0], size: 0.054519 G, files: /storage/mytopling/.rocksdb [004809] 2024/11/19-15:21:10.685135 139851158521600 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777215 2024/11/19-15:21:10.685268 139851158521600 (Original Log Time 2024/11/19-15:21:09.477466) EVENT_LOG_v1 {"time_micros": 1732000869477451, "job": 147, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4761], "score": 1.77263, "input_data_size": 54703820, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:10.685278 139851158521600 (Original Log Time 2024/11/19-15:21:10.684844) [compaction_job.cc:2022] [default] [JOB 147] Compacted 1@2 files to L3 => 54519376 bytes 2024/11/19-15:21:10.685283 139851158521600 (Original Log Time 2024/11/19-15:21:10.685102) [compaction_job.cc:1224] [default] compacted to: files[6 35 57 22 0 0 0] max score 2.79 (77 files need compaction), MB/sec: 45.3 rd, 45.2 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(52.2, 0.0 +0.0 blob) out(52.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 84154, records dropped: 0 output_compression: Snappy 2024/11/19-15:21:10.685307 139851158521600 (Original Log Time 2024/11/19-15:21:10.685128) EVENT_LOG_v1 {"time_micros": 1732000870685116, "cf": "default", "job": 147, "event": "compaction_finished", "compaction_time_micros": 1207050, "compaction_time_cpu_micros": 475306, "output_level": 3, "num_output_files": 1, "total_output_size": 54519376, "num_input_records": 84154, "num_output_records": 84154, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 35, 57, 22, 0, 0, 0]} 2024/11/19-15:21:10.685429 139851158521600 [compaction_job.cc:2437] [default] [JOB 155] Compacting 4@2 files to L3, score 1.61, subcompactions 1 : 1 2024/11/19-15:21:10.685442 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 153 Base level 2, inputs: [4779(52MB) 4780(52MB) 4781(52MB) 4782(52MB)] 2024/11/19-15:21:10.708264 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-00 2024/11/19-15:21:10.708332 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00152/att-00/compact.done) = No such file or directory: 5.467 sec on http probe failed: 4, retry 2024/11/19-15:21:10.995182 139851158521600 [dcompact_etcd.cc:1163] job-00155/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.309 sec, input = 255.634 MiB 2024/11/19-15:21:11.699150 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-02 2024/11/19-15:21:11.699227 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00142/att-02/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:21:11.797980 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-02 2024/11/19-15:21:11.798097 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00143/att-02/compact.done) = No such file or directory: 5.187 sec on http probe failed: 2, retry 2024/11/19-15:21:11.993335 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-00 2024/11/19-15:21:11.993410 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00152/att-00/compact.done) = No such file or directory: 6.748 sec on http probe failed: 5, giveup 2024/11/19-15:21:11.993428 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00152/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 63.819 sec, real 7.050 (submit 293.194 ms wait 6.748 s), input 63.819 MiB, send shutdown 2024/11/19-15:21:12.001489 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-00 2024/11/19-15:21:12.001691 139850978166528 [dcompact_etcd.cc:1773] job-00152/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:12.265595 139850978166528 [dcompact_etcd.cc:1163] job-00152/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.264 sec, input = 63.819 MiB 2024/11/19-15:21:12.352198 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-00 2024/11/19-15:21:12.352286 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00153/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:21:12.414541 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-01 2024/11/19-15:21:12.414608 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00146/att-01/compact.done) = No such file or directory: 9.026 sec on http probe failed: 3, retry 2024/11/19-15:21:12.501111 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-00 2024/11/19-15:21:12.501201 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00154/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:21:13.709279 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-02 2024/11/19-15:21:13.709365 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00142/att-02/compact.done) = No such file or directory: 11.201 sec on http probe failed: 5, giveup 2024/11/19-15:21:13.709386 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00142/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.813 sec, real 11.533 (submit 321.831 ms wait 11.201 s), input 126.813 MiB, send shutdown 2024/11/19-15:21:13.714738 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00142/att-02 2024/11/19-15:21:13.714955 139851127064320 [dcompact_etcd.cc:1773] job-00142/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:13.715184 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00142) = success 2024/11/19-15:21:13.715190 139851127064320 [dcompact_etcd.cc:1773] job-00142/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:13.715209 139851127064320 (Original Log Time 2024/11/19-15:20:39.117255) EVENT_LOG_v1 {"time_micros": 1732000839117248, "job": 142, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4707, 4708, 4709, 4710], "score": 2.16945, "input_data_size": 134099712, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:13.715216 139851127064320 [compaction_job.cc:2437] [default] [JOB 142] Compacting 4@1 files to L2, score 2.17, subcompactions 1 : 1 2024/11/19-15:21:13.715224 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 140 Base level 1, inputs: [4707(31MB) 4708(31MB) 4709(31MB) 4710(31MB)] 2024/11/19-15:21:13.715453 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004810.sst) /storage/mytopling/..//tmp/Topling-82038-Ie2ngW 2024/11/19-15:21:13.768538 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004810.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:21:13.771880 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004810.sst): index pass time = 0.00's, 156.337'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64714 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:21:13.808097 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-02 2024/11/19-15:21:13.808189 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00143/att-02/compact.done) = No such file or directory: 7.196 sec on http probe failed: 3, retry 2024/11/19-15:21:13.835491 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004810.sst): first pass time = 0.12's, 559.114'MB/sec 2024/11/19-15:21:13.835543 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004810.sst): old prealloc_size = 73819750, real_size = 66151808 2024/11/19-15:21:13.835600 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004810.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64714 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:21:13.835641 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004810.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.067 GB 2024/11/19-15:21:13.870501 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-01 2024/11/19-15:21:13.870594 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00152/att-01/compact.done) = No such file or directory: 1.596 sec on http probe failed: 1, retry 2024/11/19-15:21:13.895883 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004810.sst): second pass time = 0.00's,167894732.143'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1096.244'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2070.762'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.373'MB/sec write SST data time = 0.02's, 3925.365'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 167894732.143'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64714 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 64714 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0464 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 7.551060 GB avg = 0.276 KB (by entry num) total key len = 0.695310 GB avg = 0.025 KB total ukey len = 0.476619 GB avg = 0.017 KB total ukey num = 0.027336402 Billion total entry num = 0.027336402 Billion write speed all = 52.548844833 MB/sec (with seq num) write speed all = 51.155265531 MB/sec (without seq num) 2024/11/19-15:21:13.912071 139851127064320 [compaction_job.cc:1948] [default] [JOB 142] Generated table #4810: 64714 keys, 66137028 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:13.912124 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000873912099, "cf_name": "default", "job": 142, "event": "table_file_creation", "file_number": 4810, "file_size": 66137028, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025264, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294280, "raw_average_key_size": 20, "raw_value_size": 65814735, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64714, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000873, "oldest_key_time": 0, "file_creation_time": 1732000873, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4810, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:13.927419 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004811.sst) /storage/mytopling/..//tmp/Topling-82038-uRv3pT 2024/11/19-15:21:13.980239 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004811.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:21:13.980355 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004811.sst): index pass time = 0.00's,6268.400'MB/sec index type = UintIndex_AllOne usrkeys = 63513 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:21:14.046783 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004811.sst): first pass time = 0.12's, 551.866'MB/sec 2024/11/19-15:21:14.046838 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004811.sst): old prealloc_size = 73819750, real_size = 64816312 2024/11/19-15:21:14.046862 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004811.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63513 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0646 GB zip-val = 0.0648 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:21:14.113042 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004811.sst): second pass time = 0.00's,159489972.840'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 979.039'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.530'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.337'MB/sec write SST data time = 0.02's, 3393.126'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 159489972.840'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63513 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 63513 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0000 GB value = 0.0648 GB dict = 0.00 MB all = 0.0648 GB } UnZip/Zip{ index =3969.5625 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 7.615653 GB avg = 0.278 KB (by entry num) total key len = 0.696580 GB avg = 0.025 KB total ukey len = 0.477381 GB avg = 0.017 KB total ukey num = 0.027399915 Billion total entry num = 0.027399915 Billion write speed all = 52.895355336 MB/sec (with seq num) write speed all = 51.500468460 MB/sec (without seq num) 2024/11/19-15:21:14.194072 139851127064320 [compaction_job.cc:1948] [default] [JOB 142] Generated table #4811: 63513 keys, 64801354 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:14.194132 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000874194108, "cf_name": "default", "job": 142, "event": "table_file_creation", "file_number": 4811, "file_size": 64801354, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64800064, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1270260, "raw_average_key_size": 20, "raw_value_size": 64593439, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63513, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000873, "oldest_key_time": 0, "file_creation_time": 1732000873, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4811, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:14.209171 139851127064320 [compaction_job.cc:703] job-00142: subcompact[0], size: 0.130938 G, files: /storage/mytopling/.rocksdb [004810,004811] 2024/11/19-15:21:14.209861 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777216 2024/11/19-15:21:14.210036 139851127064320 (Original Log Time 2024/11/19-15:21:13.715254) EVENT_LOG_v1 {"time_micros": 1732000873715236, "job": 142, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4707, 4708, 4709, 4710], "score": 2.16945, "input_data_size": 134099712, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:14.210048 139851127064320 (Original Log Time 2024/11/19-15:21:14.209589) [compaction_job.cc:2022] [default] [JOB 142] Compacted 4@1 files to L2 => 130938382 bytes 2024/11/19-15:21:14.210052 139851127064320 (Original Log Time 2024/11/19-15:21:14.209828) [compaction_job.cc:1224] [default] compacted to: files[6 31 59 22 0 0 0] max score 3.15 (75 files need compaction), MB/sec: 271.5 rd, 265.1 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(124.9 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128227, records dropped: 0 output_compression: Snappy 2024/11/19-15:21:14.210075 139851127064320 (Original Log Time 2024/11/19-15:21:14.209854) EVENT_LOG_v1 {"time_micros": 1732000874209841, "cf": "default", "job": 142, "event": "compaction_finished", "compaction_time_micros": 493933, "compaction_time_cpu_micros": 398378, "output_level": 2, "num_output_files": 2, "total_output_size": 130938382, "num_input_records": 128227, "num_output_records": 128227, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 31, 59, 22, 0, 0, 0]} 2024/11/19-15:21:14.214001 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004680.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:14.214028 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000874214025, "job": 142, "event": "table_file_deletion", "file_number": 4680} 2024/11/19-15:21:14.218541 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004681.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:14.218563 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000874218560, "job": 142, "event": "table_file_deletion", "file_number": 4681} 2024/11/19-15:21:14.222528 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004682.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:14.222548 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000874222546, "job": 142, "event": "table_file_deletion", "file_number": 4682} 2024/11/19-15:21:14.226463 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004683.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:14.226483 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000874226481, "job": 142, "event": "table_file_deletion", "file_number": 4683} 2024/11/19-15:21:14.226654 139851127064320 [compaction_job.cc:2437] [default] [JOB 156] Compacting 4@2 files to L3, score 1.52, subcompactions 1 : 1 2024/11/19-15:21:14.226664 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 154 Base level 2, inputs: [4769(38MB) 4602(6826KB) 4603(31MB) 4606(62MB)] 2024/11/19-15:21:14.361461 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-00 2024/11/19-15:21:14.361532 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00153/att-00/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:21:14.420546 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-01 2024/11/19-15:21:14.420604 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00146/att-01/compact.done) = No such file or directory: 11.032 sec on http probe failed: 4, retry 2024/11/19-15:21:14.506256 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-00 2024/11/19-15:21:14.506344 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00154/att-00/compact.done) = No such file or directory: 7.005 sec on http probe failed: 2, retry 2024/11/19-15:21:14.529452 139851127064320 [dcompact_etcd.cc:1163] job-00156/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.302 sec, input = 149.879 MiB 2024/11/19-15:21:15.152937 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-01 2024/11/19-15:21:15.153062 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00152/att-01/compact.done) = No such file or directory: 2.881 sec on http probe failed: 2, retry 2024/11/19-15:21:15.814051 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-02 2024/11/19-15:21:15.814152 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00143/att-02/compact.done) = No such file or directory: 9.206 sec on http probe failed: 4, retry 2024/11/19-15:21:16.001139 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-00 2024/11/19-15:21:16.001250 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00155/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:21:16.210560 139852362282752 [db_impl_write.cc:1661] Flushing all column families with data in WAL number 4320. Total log size is 8724808169 while max_total_wal_size is 8724152320 2024/11/19-15:21:16.211112 139852362282752 [db_impl_write.cc:2181] [__system__] New memtable created with log file: #4812. Immutable memtables: 0. 2024/11/19-15:21:16.211223 139852362282752 [cspp_memtable.cc:1074] ConvertToReadOnly in MarkReadOnly, mem_size = 0.656 M, time = 9.074 us 2024/11/19-15:21:16.211318 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 157] Syncing log #4798 2024/11/19-15:21:16.211409 139850957195008 (Original Log Time 2024/11/19-15:21:16.211292) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [__system__], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:21:16.211417 139850957195008 [flush_job.cc:879] [__system__] [JOB 157] Flushing memtable with next log file: 4812 2024/11/19-15:21:16.211436 139850957195008 EVENT_LOG_v1 {"time_micros": 1732000876211430, "job": 157, "event": "flush_started", "num_memtables": 1, "num_entries": 4491, "num_deletes": 50, "total_data_size": 352999, "memory_usage": 688520, "num_range_deletes": 0, "flush_reason": "WAL Full"} 2024/11/19-15:21:16.211439 139850957195008 [flush_job.cc:910] [__system__] [JOB 157] Level-0 flush table #4813: started 2024/11/19-15:21:16.212010 139850957195008 EVENT_LOG_v1 {"time_micros": 1732000876211989, "cf_name": "__system__", "job": 157, "event": "table_file_creation", "file_number": 4813, "file_size": 7770, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 105631342, "largest_seqno": 131361268, "table_properties": {"data_size": 1344, "index_size": 5632, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1603, "raw_average_key_size": 21, "raw_value_size": 1281, "raw_average_value_size": 17, "num_data_blocks": 1, "num_entries": 74, "num_filter_entries": 0, "num_deletions": 40, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "__system__", "column_family_id": 1, "comparator": "leveldb.BytewiseComparator", "merge_operator": "Rdb_system_merge_op", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000876, "oldest_key_time": 1732000471, "file_creation_time": 1732000876, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4813, "seqno_to_time_mapping": "N/A"}} 2024/11/19-15:21:16.212436 139850957195008 [flush_job.cc:1083] [__system__] [JOB 157] Flush lasted 1029 microseconds, and 1001 cpu microseconds. 2024/11/19-15:21:16.212562 139850957195008 (Original Log Time 2024/11/19-15:21:16.212024) [flush_job.cc:1029] [__system__] [JOB 157] Level-0 flush table #4813: 7770 bytes OK (needs compaction) 2024/11/19-15:21:16.212568 139850957195008 (Original Log Time 2024/11/19-15:21:16.212445) [memtable_list.cc:560] [__system__] Level-0 commit table #4813 started 2024/11/19-15:21:16.212571 139850957195008 (Original Log Time 2024/11/19-15:21:16.212505) [memtable_list.cc:759] [__system__] Level-0 commit table #4813: memtable #1 done 2024/11/19-15:21:16.212574 139850957195008 (Original Log Time 2024/11/19-15:21:16.212518) EVENT_LOG_v1 {"time_micros": 1732000876212512, "job": 157, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:21:16.212577 139850957195008 (Original Log Time 2024/11/19-15:21:16.212540) [db_impl_compaction_flush.cc:310] [__system__] Level summary: files[1 1 0 0 0 0 0] max score 0.25 (1 files need compaction) 2024/11/19-15:21:16.212590 139850957195008 [db_impl_files.cc:470] [JOB 157] Try to delete WAL files size 8359499293, prev total WAL file size 8724808290, number of live WAL files 20. 2024/11/19-15:21:16.225633 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004320.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.240045 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004324.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.254990 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004339.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.271110 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004427.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.283222 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004431.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.295930 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004435.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.333640 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004443.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.369459 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-00 2024/11/19-15:21:16.369537 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00153/att-00/compact.done) = No such file or directory: 9.020 sec on http probe failed: 3, retry 2024/11/19-15:21:16.387760 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004573.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.430122 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-01 2024/11/19-15:21:16.430190 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00146/att-01/compact.done) = No such file or directory: 13.038 sec on http probe failed: 5, giveup 2024/11/19-15:21:16.430212 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00146/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 254.479 sec, real 13.318 (submit 270.481 ms wait 13.038 s), input 254.479 MiB, send shutdown 2024/11/19-15:21:16.437319 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-01 2024/11/19-15:21:16.437551 139851137550080 [dcompact_etcd.cc:1773] job-00146/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:16.437967 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-01 2024/11/19-15:21:16.438039 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00152/att-01/compact.done) = No such file or directory: 4.164 sec on http probe failed: 3, retry 2024/11/19-15:21:16.440266 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004578.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.490064 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004585.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.515081 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-00 2024/11/19-15:21:16.515141 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00154/att-00/compact.done) = No such file or directory: 9.010 sec on http probe failed: 3, retry 2024/11/19-15:21:16.544484 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004595.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.594419 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004604.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.645448 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004626.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.701202 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004745.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.727991 139851137550080 [dcompact_etcd.cc:1163] job-00146/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.290 sec, input = 254.479 MiB 2024/11/19-15:21:16.765242 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004758.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.821407 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004767.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.878145 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004776.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:16.935420 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004785.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:17.723916 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-01 2024/11/19-15:21:17.724030 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00152/att-01/compact.done) = No such file or directory: 5.449 sec on http probe failed: 4, retry 2024/11/19-15:21:17.823636 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-02 2024/11/19-15:21:17.823711 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00143/att-02/compact.done) = No such file or directory: 11.212 sec on http probe failed: 5, giveup 2024/11/19-15:21:17.823729 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00143/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.830 sec, real 11.483 (submit 262.123 ms wait 11.212 s), input 126.830 MiB, send shutdown 2024/11/19-15:21:17.828707 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00143/att-02 2024/11/19-15:21:17.828933 139851148035840 [dcompact_etcd.cc:1773] job-00143/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:17.829156 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00143) = success 2024/11/19-15:21:17.829162 139851148035840 [dcompact_etcd.cc:1773] job-00143/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:17.829181 139851148035840 (Original Log Time 2024/11/19-15:20:43.354867) EVENT_LOG_v1 {"time_micros": 1732000843354860, "job": 143, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4711, 4712, 4713, 4714], "score": 1.91967, "input_data_size": 134099264, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:17.829188 139851148035840 [compaction_job.cc:2437] [default] [JOB 143] Compacting 4@1 files to L2, score 1.92, subcompactions 1 : 1 2024/11/19-15:21:17.829196 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 141 Base level 1, inputs: [4711(31MB) 4712(31MB) 4713(31MB) 4714(31MB)] 2024/11/19-15:21:17.829414 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004814.sst) /storage/mytopling/..//tmp/Topling-82038-GbwAqe 2024/11/19-15:21:17.886016 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004814.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:21:17.889388 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004814.sst): index pass time = 0.00's, 155.040'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64712 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:21:17.959675 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004814.sst): first pass time = 0.13's, 515.242'MB/sec 2024/11/19-15:21:17.959727 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004814.sst): old prealloc_size = 73819750, real_size = 66151904 2024/11/19-15:21:17.959784 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004814.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64712 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.04 avg-zval =1020.30 2024/11/19-15:21:17.959833 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004814.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.074 sec, Key+Value = 0.067 GB 2024/11/19-15:21:18.006495 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-00 2024/11/19-15:21:18.006585 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00155/att-00/compact.done) = No such file or directory: 7.006 sec on http probe failed: 2, retry 2024/11/19-15:21:18.034944 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004814.sst): second pass time = 0.00's,205671328.125'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.08's, 879.697'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2125.147'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.333'MB/sec write SST data time = 0.03's, 2496.159'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 205671328.125'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64712 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.04 avg-zval = 1020.30 usrkeys = 64712 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.04 avg-zval = 1020.30 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0462 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 7.681468 GB avg = 0.280 KB (by entry num) total key len = 0.697875 GB avg = 0.025 KB total ukey len = 0.478158 GB avg = 0.017 KB total ukey num = 0.027464627 Billion total entry num = 0.027464627 Billion write speed all = 52.024038334 MB/sec (with seq num) write speed all = 50.659901962 MB/sec (without seq num) 2024/11/19-15:21:18.041163 139851148035840 [compaction_job.cc:1948] [default] [JOB 143] Generated table #4814: 64712 keys, 66137156 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.041213 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878041190, "cf_name": "default", "job": 143, "event": "table_file_creation", "file_number": 4814, "file_size": 66137156, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025360, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294240, "raw_average_key_size": 20, "raw_value_size": 65814825, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64712, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000877, "oldest_key_time": 0, "file_creation_time": 1732000877, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4814, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:18.059824 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004815.sst) /storage/mytopling/..//tmp/Topling-82038-Ltzlyc 2024/11/19-15:21:18.116116 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004815.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:21:18.121208 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004815.sst): index pass time = 0.01's, 100.439'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63533 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:21:18.189170 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004815.sst): first pass time = 0.13's, 509.470'MB/sec 2024/11/19-15:21:18.189224 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004815.sst): old prealloc_size = 73819750, real_size = 64942288 2024/11/19-15:21:18.189279 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004815.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63533 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0646 GB zip-val = 0.0648 GB avg-val =1016.97 avg-zval =1020.22 2024/11/19-15:21:18.189330 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004815.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.073 sec, Key+Value = 0.066 GB 2024/11/19-15:21:18.252091 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004815.sst): second pass time = 0.00's,84459108.497'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1033.071'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2173.454'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.361'MB/sec write SST data time = 0.02's, 3461.788'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 84459108.497'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63533 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.97 avg-zval = 1020.22 usrkeys = 63533 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.97 avg-zval = 1020.22 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7916 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0648 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index = 7.0404 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 7.746079 GB avg = 0.281 KB (by entry num) total key len = 0.699145 GB avg = 0.025 KB total ukey len = 0.478920 GB avg = 0.017 KB total ukey num = 0.027528160 Billion total entry num = 0.027528160 Billion write speed all = 52.362473179 MB/sec (with seq num) write speed all = 50.997022219 MB/sec (without seq num) 2024/11/19-15:21:18.256737 139851148035840 [compaction_job.cc:1948] [default] [JOB 143] Generated table #4815: 63533 keys, 64927748 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.256787 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878256764, "cf_name": "default", "job": 143, "event": "table_file_creation", "file_number": 4815, "file_size": 64927748, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64817920, "index_size": 108288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1270660, "raw_average_key_size": 20, "raw_value_size": 64611218, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63533, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4815, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:18.272342 139851148035840 [compaction_job.cc:703] job-00143: subcompact[0], size: 0.131065 G, files: /storage/mytopling/.rocksdb [004814,004815] 2024/11/19-15:21:18.273120 139851148035840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777216 2024/11/19-15:21:18.273244 139851148035840 (Original Log Time 2024/11/19-15:21:17.829227) EVENT_LOG_v1 {"time_micros": 1732000877829207, "job": 143, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4711, 4712, 4713, 4714], "score": 1.91967, "input_data_size": 134099264, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:18.273251 139851148035840 (Original Log Time 2024/11/19-15:21:18.272851) [compaction_job.cc:2022] [default] [JOB 143] Compacted 4@1 files to L2 => 131064904 bytes 2024/11/19-15:21:18.273254 139851148035840 (Original Log Time 2024/11/19-15:21:18.273091) [compaction_job.cc:1224] [default] compacted to: files[6 27 61 22 0 0 0] max score 3.62 (73 files need compaction), MB/sec: 302.6 rd, 295.8 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128245, records dropped: 0 output_compression: Snappy 2024/11/19-15:21:18.273271 139851148035840 (Original Log Time 2024/11/19-15:21:18.273113) EVENT_LOG_v1 {"time_micros": 1732000878273103, "cf": "default", "job": 143, "event": "compaction_finished", "compaction_time_micros": 443128, "compaction_time_cpu_micros": 441000, "output_level": 2, "num_output_files": 2, "total_output_size": 131064904, "num_input_records": 128245, "num_output_records": 128245, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 27, 61, 22, 0, 0, 0]} 2024/11/19-15:21:18.277154 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004686.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:18.277179 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878277176, "job": 143, "event": "table_file_deletion", "file_number": 4686} 2024/11/19-15:21:18.281092 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004688.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:18.281114 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878281112, "job": 143, "event": "table_file_deletion", "file_number": 4688} 2024/11/19-15:21:18.285144 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004689.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:18.285164 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878285162, "job": 143, "event": "table_file_deletion", "file_number": 4689} 2024/11/19-15:21:18.286252 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-00 2024/11/19-15:21:18.286329 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00156/att-00/compact.done) = No such file or directory: 3.747 sec on http probe failed: 1, retry 2024/11/19-15:21:18.289054 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004690.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:18.289075 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878289073, "job": 143, "event": "table_file_deletion", "file_number": 4690} 2024/11/19-15:21:18.289240 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004742.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:18.289249 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878289247, "job": 143, "event": "table_file_deletion", "file_number": 4742} 2024/11/19-15:21:18.289381 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004743.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:18.289390 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878289388, "job": 143, "event": "table_file_deletion", "file_number": 4743} 2024/11/19-15:21:18.300139 139851148035840 [compaction_job.cc:2437] [default] [JOB 158] Compacting 6@0 + 27@1 files to L1, score 3.62, subcompactions 3 : 3 2024/11/19-15:21:18.300161 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 156 Base level 0, inputs: [4799(513MB) 4786(512MB) 4777(521MB) 4768(516MB) 4759(516MB) 4746(515MB)], [4715(31MB) 4716(31MB) 4717(31MB) 4718(31MB) 4719(31MB) 4720(31MB) 4721(31MB) 4722(31MB) 4723(31MB) 4724(31MB) 4725(31MB) 4726(31MB) 4727(31MB) 4728(31MB) 4729(31MB) 4730(31MB) 4731(31MB) 4732(31MB) 4733(31MB) 4734(31MB) 4735(31MB) 4736(31MB) 4737(31MB) 4738(31MB) 4739(31MB) 4740(31MB) 4741(23MB)] 2024/11/19-15:21:18.300675 139851148035840 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4816: 16 keys, 6067 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.300709 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878300690, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4816, "file_size": 6067, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 122618538, "largest_seqno": 129745041, "table_properties": {"data_size": 1920, "index_size": 3328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 256, "raw_average_key_size": 16, "raw_value_size": 1876, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 16, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4816, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.301362 139851148035840 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4819: 167 keys, 28920 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.301392 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878301376, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4819, "file_size": 28920, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121930472, "largest_seqno": 130124230, "table_properties": {"data_size": 20224, "index_size": 7872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3340, "raw_average_key_size": 20, "raw_value_size": 20220, "raw_average_value_size": 121, "num_data_blocks": 1, "num_entries": 167, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4819, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.333777 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4818: 32153 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.333846 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000878333816, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4818, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 117782982, "largest_seqno": 117851147, "table_properties": {"data_size": 32697216, "index_size": 826688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643060, "raw_average_key_size": 20, "raw_value_size": 32697203, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32153, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4818, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.340171 139851148035840 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4820: 49845 keys, 33506688 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.340212 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878340192, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4820, "file_size": 33506688, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121930474, "largest_seqno": 122547794, "table_properties": {"data_size": 32158976, "index_size": 1346880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196280, "raw_average_key_size": 24, "raw_value_size": 32158938, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49845, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4820, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.366298 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4821: 32053 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.366340 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000878366320, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4821, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 117851148, "largest_seqno": 117925849, "table_properties": {"data_size": 32599040, "index_size": 924608, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32598977, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4821, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.374664 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-00 2024/11/19-15:21:18.374750 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00153/att-00/compact.done) = No such file or directory: 11.028 sec on http probe failed: 4, retry 2024/11/19-15:21:18.379288 139851148035840 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4822: 49863 keys, 33506944 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.379333 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878379311, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4822, "file_size": 33506944, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 122540638, "largest_seqno": 123504689, "table_properties": {"data_size": 32157184, "index_size": 1348928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196712, "raw_average_key_size": 24, "raw_value_size": 32157153, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49863, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4822, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.398483 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4823: 32052 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.398527 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000878398507, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4823, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 117925850, "largest_seqno": 118593748, "table_properties": {"data_size": 32597760, "index_size": 925952, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641040, "raw_average_key_size": 20, "raw_value_size": 32597760, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32052, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4823, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.406587 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4817: 289892 keys, 22797635 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.406645 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000878406625, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4817, "file_size": 22797635, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125337015, "largest_seqno": 130136235, "table_properties": {"data_size": 15652736, "index_size": 7144064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 5797840, "raw_average_key_size": 20, "raw_value_size": 15652725, "raw_average_value_size": 53, "num_data_blocks": 1, "num_entries": 289892, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4817, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.418658 139851148035840 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4824: 49849 keys, 33506624 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.418702 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878418681, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4824, "file_size": 33506624, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 123337415, "largest_seqno": 124500434, "table_properties": {"data_size": 32158784, "index_size": 1347008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196376, "raw_average_key_size": 24, "raw_value_size": 32158775, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49849, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4824, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.431011 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4825: 32088 keys, 33524800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.431053 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000878431033, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4825, "file_size": 33524800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 118593749, "largest_seqno": 118668620, "table_properties": {"data_size": 32633088, "index_size": 890880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641760, "raw_average_key_size": 20, "raw_value_size": 32633049, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32088, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4825, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.457507 139851148035840 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4827: 49825 keys, 33506560 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.457558 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878457535, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4827, "file_size": 33506560, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 124017761, "largest_seqno": 125334286, "table_properties": {"data_size": 32158656, "index_size": 1347072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1195800, "raw_average_key_size": 24, "raw_value_size": 32158640, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49825, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4827, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.463689 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4828: 32150 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.463734 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000878463714, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4828, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 118668621, "largest_seqno": 118771421, "table_properties": {"data_size": 32696192, "index_size": 827456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32696149, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4828, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.485140 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4826: 149400 keys, 3902030 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.485186 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000878485166, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4826, "file_size": 3902030, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121919568, "largest_seqno": 130124228, "table_properties": {"data_size": 149440, "index_size": 3751744, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3585600, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 149400, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4826, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.495607 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4830: 32058 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.495652 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000878495632, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4830, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 118771422, "largest_seqno": 118833489, "table_properties": {"data_size": 32602048, "index_size": 921664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32601994, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4830, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.496282 139851148035840 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4829: 49858 keys, 33506880 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.496322 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878496303, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4829, "file_size": 33506880, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125019326, "largest_seqno": 126060803, "table_properties": {"data_size": 32159424, "index_size": 1346624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196592, "raw_average_key_size": 24, "raw_value_size": 32159403, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49858, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4829, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.523485 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-00 2024/11/19-15:21:18.523598 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00154/att-00/compact.done) = No such file or directory: 11.019 sec on http probe failed: 4, retry 2024/11/19-15:21:18.537467 139851148035840 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4833: 49843 keys, 33506816 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.537525 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878537501, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4833, "file_size": 33506816, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126020536, "largest_seqno": 126822624, "table_properties": {"data_size": 32159232, "index_size": 1346752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196232, "raw_average_key_size": 24, "raw_value_size": 32159190, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49843, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4833, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.543119 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4832: 32056 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.543187 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000878543156, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4832, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 118776894, "largest_seqno": 118836901, "table_properties": {"data_size": 32604032, "index_size": 920192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32603983, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4832, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.575867 139851148035840 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4834: 49863 keys, 33506688 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.575930 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878575900, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4834, "file_size": 33506688, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126822625, "largest_seqno": 127569085, "table_properties": {"data_size": 32157952, "index_size": 1347904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196712, "raw_average_key_size": 24, "raw_value_size": 32157894, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49863, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4834, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.592873 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4835: 32151 keys, 33524288 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.592950 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000878592912, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4835, "file_size": 33524288, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 118825544, "largest_seqno": 118916813, "table_properties": {"data_size": 32696960, "index_size": 826496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32696906, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4835, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.614838 139851148035840 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4836: 49826 keys, 33506688 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.614894 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878614871, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4836, "file_size": 33506688, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127569086, "largest_seqno": 128495497, "table_properties": {"data_size": 32159232, "index_size": 1346624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1195824, "raw_average_key_size": 24, "raw_value_size": 32159219, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49826, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4836, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.643316 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4837: 32058 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.643389 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000878643357, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4837, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 118916814, "largest_seqno": 119023260, "table_properties": {"data_size": 32602368, "index_size": 921280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602322, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4837, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.653497 139851148035840 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4838: 49843 keys, 33507136 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.653548 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878653527, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4838, "file_size": 33507136, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128250275, "largest_seqno": 129527763, "table_properties": {"data_size": 32158400, "index_size": 1347904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196232, "raw_average_key_size": 24, "raw_value_size": 32158352, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49843, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4838, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.662387 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4831: 504000 keys, 22330755 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.662439 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000878662418, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4831, "file_size": 22330755, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121866423, "largest_seqno": 130142237, "table_properties": {"data_size": 9979200, "index_size": 12350720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12096000, "raw_average_key_size": 24, "raw_value_size": 9979200, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 504000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4831, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.691639 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4839: 32058 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.691710 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000878691678, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4839, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119023261, "largest_seqno": 119656760, "table_properties": {"data_size": 32602624, "index_size": 920960, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602621, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4839, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.692071 139851148035840 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4840: 49864 keys, 33506624 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.692112 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878692092, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4840, "file_size": 33506624, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128974122, "largest_seqno": 130127712, "table_properties": {"data_size": 32155904, "index_size": 1349888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196736, "raw_average_key_size": 24, "raw_value_size": 32155862, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49864, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4840, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.694361 139851148035840 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4843: 2521 keys, 1699643 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.694395 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878694377, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4843, "file_size": 1699643, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130127713, "largest_seqno": 130130233, "table_properties": {"data_size": 1624256, "index_size": 74560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 60504, "raw_average_key_size": 24, "raw_value_size": 1624209, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 2521, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4843, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.751732 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4842: 32124 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.751807 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000878751777, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4842, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119656761, "largest_seqno": 119731502, "table_properties": {"data_size": 32668224, "index_size": 855360, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642480, "raw_average_key_size": 20, "raw_value_size": 32668188, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32124, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4842, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.781094 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4841: 231231 keys, 33325123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.781151 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000878781128, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4841, "file_size": 33325123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121879425, "largest_seqno": 122449561, "table_properties": {"data_size": 27032192, "index_size": 6292096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6474468, "raw_average_key_size": 28, "raw_value_size": 27032182, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231231, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4841, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.791768 139851148035840 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4844: 214108 keys, 16877891 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.791813 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000878791793, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4844, "file_size": 16877891, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121863421, "largest_seqno": 125337014, "table_properties": {"data_size": 11559872, "index_size": 5317184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4282160, "raw_average_key_size": 20, "raw_value_size": 11559859, "raw_average_value_size": 53, "num_data_blocks": 1, "num_entries": 214108, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4844, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.810865 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4845: 32058 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.810930 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000878810895, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4845, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119731503, "largest_seqno": 119814656, "table_properties": {"data_size": 32602688, "index_size": 921024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602681, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4845, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.861561 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4847: 32059 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.861669 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000878861624, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4847, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119775559, "largest_seqno": 119835566, "table_properties": {"data_size": 32603712, "index_size": 920640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603705, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4847, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.923383 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4846: 231479 keys, 33324867 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.923447 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000878923419, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4846, "file_size": 33324867, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121998400, "largest_seqno": 122617635, "table_properties": {"data_size": 27026752, "index_size": 6297280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481412, "raw_average_key_size": 28, "raw_value_size": 27026742, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231479, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4846, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.923482 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4848: 32149 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.923530 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000878923510, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4848, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119816717, "largest_seqno": 119898930, "table_properties": {"data_size": 32697216, "index_size": 826432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642980, "raw_average_key_size": 20, "raw_value_size": 32697188, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32149, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4848, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:18.970313 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4850: 32056 keys, 33524352 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:18.970370 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000878970347, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4850, "file_size": 33524352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119898931, "largest_seqno": 119976699, "table_properties": {"data_size": 32601856, "index_size": 921664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32601820, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4850, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.004161 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4851: 32058 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.004214 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000879004192, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4851, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 119976700, "largest_seqno": 120641006, "table_properties": {"data_size": 32602240, "index_size": 921920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602182, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4851, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.009251 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-01 2024/11/19-15:21:19.009351 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00152/att-01/compact.done) = No such file or directory: 6.735 sec on http probe failed: 5, giveup 2024/11/19-15:21:19.009369 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00152/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 63.819 sec, real 7.008 (submit 263.820 ms wait 6.735 s), input 63.819 MiB, send shutdown 2024/11/19-15:21:19.033630 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-01 2024/11/19-15:21:19.041953 139850978166528 [dcompact_etcd.cc:1773] job-00152/att-01: CleanFiles 0.008 sec : 0.000 2024/11/19-15:21:19.052232 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4849: 231448 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.052282 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000879052260, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4849, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 122410311, "largest_seqno": 123071319, "table_properties": {"data_size": 27024256, "index_size": 6299968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480544, "raw_average_key_size": 28, "raw_value_size": 27024244, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231448, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000878, "oldest_key_time": 0, "file_creation_time": 1732000878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4849, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.052499 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4852: 32059 keys, 33524800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.052542 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000879052522, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4852, "file_size": 33524800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120641007, "largest_seqno": 120718989, "table_properties": {"data_size": 32604992, "index_size": 918976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32604959, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4852, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.100577 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4854: 32053 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.100631 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000879100610, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4854, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120718990, "largest_seqno": 120805011, "table_properties": {"data_size": 32598656, "index_size": 924992, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32598652, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4854, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.142747 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4855: 32055 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.142799 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000879142777, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4855, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120773480, "largest_seqno": 120833487, "table_properties": {"data_size": 32599808, "index_size": 923904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32599754, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4855, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.254058 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4853: 231378 keys, 33325123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.254115 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000879254093, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4853, "file_size": 33325123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 123035904, "largest_seqno": 123574623, "table_properties": {"data_size": 27030720, "index_size": 6293568, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6478584, "raw_average_key_size": 28, "raw_value_size": 27030684, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231378, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4853, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.274944 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4856: 32067 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.275003 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000879274980, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4856, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120797065, "largest_seqno": 120858144, "table_properties": {"data_size": 32612480, "index_size": 911936, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641340, "raw_average_key_size": 20, "raw_value_size": 32612446, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32067, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4856, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.296449 139850978166528 [dcompact_etcd.cc:1163] job-00152/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.254 sec, input = 63.819 MiB 2024/11/19-15:21:19.306941 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4858: 32054 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.306994 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000879306972, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4858, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120858145, "largest_seqno": 120926853, "table_properties": {"data_size": 32598528, "index_size": 925312, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641080, "raw_average_key_size": 20, "raw_value_size": 32598486, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4858, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.374087 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4859: 32056 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.374145 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000879374122, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4859, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 120926854, "largest_seqno": 121636528, "table_properties": {"data_size": 32600896, "index_size": 923456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32600834, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4859, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.470370 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4857: 231322 keys, 33325123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.470429 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000879470406, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4857, "file_size": 33325123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 123061790, "largest_seqno": 123672825, "table_properties": {"data_size": 27030336, "index_size": 6293952, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6477016, "raw_average_key_size": 28, "raw_value_size": 27030276, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231322, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4857, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.470584 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4860: 32085 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.470631 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000879470610, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4860, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121636529, "largest_seqno": 121741338, "table_properties": {"data_size": 32628352, "index_size": 895552, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641700, "raw_average_key_size": 20, "raw_value_size": 32628309, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32085, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4860, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.502286 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4862: 32054 keys, 33524288 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.502339 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000879502318, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4862, "file_size": 33524288, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121741339, "largest_seqno": 121794303, "table_properties": {"data_size": 32599296, "index_size": 924160, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641080, "raw_average_key_size": 20, "raw_value_size": 32599282, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4862, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.543556 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4863: 32057 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.543612 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000879543590, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4863, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121760401, "largest_seqno": 121840411, "table_properties": {"data_size": 32600960, "index_size": 923520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32600940, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4863, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.593584 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4861: 231470 keys, 33324867 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.593637 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000879593615, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4861, "file_size": 33324867, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 123672826, "largest_seqno": 124312444, "table_properties": {"data_size": 27025280, "index_size": 6298752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481160, "raw_average_key_size": 28, "raw_value_size": 27025264, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231470, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4861, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.593691 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4864: 32114 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.593737 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000879593717, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4864, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121766356, "largest_seqno": 121828479, "table_properties": {"data_size": 32659904, "index_size": 864512, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642280, "raw_average_key_size": 20, "raw_value_size": 32659858, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32114, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4864, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.635170 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4866: 32056 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.635227 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000879635204, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4866, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121828480, "largest_seqno": 121909691, "table_properties": {"data_size": 32601408, "index_size": 922176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32601368, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4866, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.680315 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4867: 32058 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.680376 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000879680353, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4867, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121909692, "largest_seqno": 122629726, "table_properties": {"data_size": 32601088, "index_size": 922624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32601047, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4867, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.712527 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4865: 231490 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.712592 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000879712570, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4865, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 124033212, "largest_seqno": 124572421, "table_properties": {"data_size": 27027072, "index_size": 6297088, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481720, "raw_average_key_size": 28, "raw_value_size": 27027024, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231490, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4865, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.723826 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4868: 32151 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.723869 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000879723850, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4868, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 122629727, "largest_seqno": 122701799, "table_properties": {"data_size": 32697088, "index_size": 827200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32697031, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4868, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.812705 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4870: 32053 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.812763 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000879812741, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4870, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 122701800, "largest_seqno": 122774734, "table_properties": {"data_size": 32596416, "index_size": 927488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32596368, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4870, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.821516 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4869: 231271 keys, 33325187 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.821568 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000879821548, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4869, "file_size": 33325187, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 124353973, "largest_seqno": 124934998, "table_properties": {"data_size": 27032832, "index_size": 6291520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6475588, "raw_average_key_size": 28, "raw_value_size": 27032778, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231271, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4869, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:19.944509 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4871: 32126 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:19.944568 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000879944545, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4871, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 122774735, "largest_seqno": 122826868, "table_properties": {"data_size": 32671424, "index_size": 852352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642520, "raw_average_key_size": 20, "raw_value_size": 32671369, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32126, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4871, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.016784 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-00 2024/11/19-15:21:20.016866 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00155/att-00/compact.done) = No such file or directory: 9.011 sec on http probe failed: 3, retry 2024/11/19-15:21:20.047208 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4872: 231430 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.047260 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000880047238, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4872, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 124934999, "largest_seqno": 125524521, "table_properties": {"data_size": 27026880, "index_size": 6297280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480040, "raw_average_key_size": 28, "raw_value_size": 27026860, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231430, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4872, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.148564 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4873: 32056 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.148619 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000880148595, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4873, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 122758346, "largest_seqno": 122828926, "table_properties": {"data_size": 32603456, "index_size": 920640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32603433, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000879, "oldest_key_time": 0, "file_creation_time": 1732000879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4873, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.152645 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4874: 231455 keys, 33324931 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.152694 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000880152672, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4874, "file_size": 33324931, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125062112, "largest_seqno": 125606418, "table_properties": {"data_size": 27024256, "index_size": 6299840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480740, "raw_average_key_size": 28, "raw_value_size": 27024250, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231455, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4874, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.180680 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4875: 32139 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.180732 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000880180710, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4875, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 122828927, "largest_seqno": 122903886, "table_properties": {"data_size": 32686464, "index_size": 837248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642780, "raw_average_key_size": 20, "raw_value_size": 32686421, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32139, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4875, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.213568 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4877: 32059 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.213619 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000880213598, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4877, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 122903887, "largest_seqno": 122981953, "table_properties": {"data_size": 32603520, "index_size": 920576, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603483, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4877, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.246350 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4878: 32059 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.246403 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000880246381, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4878, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 122981954, "largest_seqno": 123651269, "table_properties": {"data_size": 32604032, "index_size": 920384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603984, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4878, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.256856 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4876: 231411 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.256904 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000880256884, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4876, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125606419, "largest_seqno": 126136017, "table_properties": {"data_size": 27029248, "index_size": 6294976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6479508, "raw_average_key_size": 28, "raw_value_size": 27029234, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231411, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4876, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.292719 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-00 2024/11/19-15:21:20.292803 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00156/att-00/compact.done) = No such file or directory: 5.757 sec on http probe failed: 2, retry 2024/11/19-15:21:20.361991 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4879: 32151 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.362047 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000880362022, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4879, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 123651270, "largest_seqno": 123723401, "table_properties": {"data_size": 32697152, "index_size": 826624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32697099, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4879, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.380135 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-00 2024/11/19-15:21:20.380213 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00153/att-00/compact.done) = No such file or directory: 13.033 sec on http probe failed: 5, giveup 2024/11/19-15:21:20.380232 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00153/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 229.044 sec, real 13.345 (submit 306.527 ms wait 13.033 s), input 229.044 MiB, send shutdown 2024/11/19-15:21:20.388060 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-00 2024/11/19-15:21:20.416404 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4880: 231340 keys, 33325123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.416450 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000880416429, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4880, "file_size": 33325123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126063565, "largest_seqno": 126538628, "table_properties": {"data_size": 27030528, "index_size": 6293760, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6477520, "raw_average_key_size": 28, "raw_value_size": 27030528, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231340, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4880, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.416499 139851169007360 [dcompact_etcd.cc:1773] job-00153/att-00: CleanFiles 0.028 sec : 0.000 2024/11/19-15:21:20.416598 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4881: 32056 keys, 33524352 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.416645 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000880416619, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4881, "file_size": 33524352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 123723402, "largest_seqno": 123816518, "table_properties": {"data_size": 32602112, "index_size": 921408, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32602106, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4881, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.449089 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4883: 32058 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.449136 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000880449115, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4883, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 123765477, "largest_seqno": 123825485, "table_properties": {"data_size": 32603584, "index_size": 920000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603522, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4883, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.491519 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4884: 32102 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.491573 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000880491551, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4884, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 123808575, "largest_seqno": 123889410, "table_properties": {"data_size": 32646592, "index_size": 877696, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642040, "raw_average_key_size": 20, "raw_value_size": 32646592, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32102, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4884, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.532544 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-00 2024/11/19-15:21:20.532680 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00154/att-00/compact.done) = No such file or directory: 13.028 sec on http probe failed: 5, giveup 2024/11/19-15:21:20.532716 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00154/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 233.065 sec, real 13.333 (submit 296.176 ms wait 13.028 s), input 233.065 MiB, send shutdown 2024/11/19-15:21:20.537615 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-00 2024/11/19-15:21:20.537985 139851179493120 [dcompact_etcd.cc:1773] job-00154/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:20.543737 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4882: 231478 keys, 33324931 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.543810 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000880543775, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4882, "file_size": 33324931, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126241211, "largest_seqno": 126858578, "table_properties": {"data_size": 27025024, "index_size": 6299072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481384, "raw_average_key_size": 28, "raw_value_size": 27024980, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231478, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4882, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.547678 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4885: 32057 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.547758 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000880547717, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4885, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 123889411, "largest_seqno": 123968598, "table_properties": {"data_size": 32602880, "index_size": 921280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602821, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4885, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.619110 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4887: 32057 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.619212 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000880619171, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4887, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 123968599, "largest_seqno": 124632634, "table_properties": {"data_size": 32603008, "index_size": 920896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602986, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4887, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.673790 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4888: 32142 keys, 33524800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.673848 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000880673825, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4888, "file_size": 33524800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 124632635, "largest_seqno": 124714404, "table_properties": {"data_size": 32687936, "index_size": 836032, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642840, "raw_average_key_size": 20, "raw_value_size": 32687893, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32142, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4888, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.694120 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4886: 231486 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.694174 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000880694153, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4886, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126858579, "largest_seqno": 127427439, "table_properties": {"data_size": 27027520, "index_size": 6296640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481608, "raw_average_key_size": 28, "raw_value_size": 27027504, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231486, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4886, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.721624 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4889: 32057 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.721676 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000880721654, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4889, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 124714405, "largest_seqno": 124797156, "table_properties": {"data_size": 32603200, "index_size": 921216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32603160, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4889, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.724323 139851169007360 [dcompact_etcd.cc:1163] job-00153/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.308 sec, input = 229.044 MiB 2024/11/19-15:21:20.801993 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4891: 32053 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.802049 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000880802026, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4891, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 124763645, "largest_seqno": 124823653, "table_properties": {"data_size": 32598400, "index_size": 926080, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32598359, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4891, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.802182 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4890: 231264 keys, 33325123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.802227 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000880802207, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4890, "file_size": 33325123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127059128, "largest_seqno": 127536405, "table_properties": {"data_size": 27031424, "index_size": 6292864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6475392, "raw_average_key_size": 28, "raw_value_size": 27031420, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231264, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4890, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.828493 139851179493120 [dcompact_etcd.cc:1163] job-00154/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.290 sec, input = 233.065 MiB 2024/11/19-15:21:20.843955 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4892: 32151 keys, 33524288 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.844022 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000880843994, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4892, "file_size": 33524288, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 124789207, "largest_seqno": 124850371, "table_properties": {"data_size": 32696832, "index_size": 826624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32696832, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4892, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.900868 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-02 2024/11/19-15:21:20.900977 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00152/att-02/compact.done) = No such file or directory: 1.596 sec on http probe failed: 1, retry 2024/11/19-15:21:20.970541 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4894: 32054 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.970602 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000880970579, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4894, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 124850372, "largest_seqno": 124953284, "table_properties": {"data_size": 32598976, "index_size": 924864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641080, "raw_average_key_size": 20, "raw_value_size": 32598939, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4894, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:20.995266 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4893: 231417 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:20.995327 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000880995304, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4893, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127480068, "largest_seqno": 128070889, "table_properties": {"data_size": 27027328, "index_size": 6296832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6479676, "raw_average_key_size": 28, "raw_value_size": 27027326, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231417, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4893, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.002420 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4895: 32056 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.002467 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000881002446, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4895, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 124953285, "largest_seqno": 125592851, "table_properties": {"data_size": 32601408, "index_size": 923008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32601360, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4895, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.118471 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4897: 32093 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.118531 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000881118508, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4897, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125592852, "largest_seqno": 125695751, "table_properties": {"data_size": 32639936, "index_size": 883712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641860, "raw_average_key_size": 20, "raw_value_size": 32639911, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32093, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4897, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.167393 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4896: 231463 keys, 33324931 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.167451 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000881167428, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4896, "file_size": 33324931, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128025029, "largest_seqno": 128565653, "table_properties": {"data_size": 27024384, "index_size": 6299712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480964, "raw_average_key_size": 28, "raw_value_size": 27024382, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231463, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000880, "oldest_key_time": 0, "file_creation_time": 1732000880, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4896, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.167491 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4898: 32057 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.167550 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000881167522, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4898, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125695752, "largest_seqno": 125777834, "table_properties": {"data_size": 32600512, "index_size": 923584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32600492, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4898, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.207505 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4900: 32057 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.207614 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000881207567, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4900, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125761863, "largest_seqno": 125821870, "table_properties": {"data_size": 32601152, "index_size": 922944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601108, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4900, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.272548 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4901: 32131 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.272635 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000881272601, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4901, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125769890, "largest_seqno": 125841036, "table_properties": {"data_size": 32677376, "index_size": 846656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642620, "raw_average_key_size": 20, "raw_value_size": 32677363, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32131, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4901, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.294632 139852349699840 [db_impl_write.cc:2181] [default] New memtable created with log file: #4903. Immutable memtables: 0. 2024/11/19-15:21:21.295342 139852349699840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16273899 2024/11/19-15:21:21.295527 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 159] Syncing log #4812 2024/11/19-15:21:21.296784 139850967680768 (Original Log Time 2024/11/19-15:21:21.295506) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:21:21.296801 139850967680768 [flush_job.cc:879] [default] [JOB 159] Flushing memtable with next log file: 4903 2024/11/19-15:21:21.296837 139850967680768 EVENT_LOG_v1 {"time_micros": 1732000881296826, "job": 159, "event": "flush_started", "num_memtables": 1, "num_entries": 1527996, "num_deletes": 0, "total_data_size": 477835376, "memory_usage": 546831752, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:21:21.296844 139850967680768 [flush_job.cc:910] [default] [JOB 159] Level-0 flush table #4904: started 2024/11/19-15:21:21.297633 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000018.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004904.sst in ConvertToSST, mem_size = 521.499 M, time = 769.664 us 2024/11/19-15:21:21.333246 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4899: 231387 keys, 33325187 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.333341 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000881333301, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4899, "file_size": 33325187, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128111088, "largest_seqno": 128672140, "table_properties": {"data_size": 27029696, "index_size": 6294656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6478836, "raw_average_key_size": 28, "raw_value_size": 27029650, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231387, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4899, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.344148 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4902: 32057 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.344219 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000881344187, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4902, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125841037, "largest_seqno": 125949888, "table_properties": {"data_size": 32602240, "index_size": 921984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602177, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4902, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.474776 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4906: 32052 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.474871 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000881474834, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4906, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125949889, "largest_seqno": 126574818, "table_properties": {"data_size": 32596992, "index_size": 926848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641040, "raw_average_key_size": 20, "raw_value_size": 32596934, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32052, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4906, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.510493 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4905: 231321 keys, 33325123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.510586 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000881510550, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4905, "file_size": 33325123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128672141, "largest_seqno": 129321424, "table_properties": {"data_size": 27030912, "index_size": 6293376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6476988, "raw_average_key_size": 28, "raw_value_size": 27030894, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231321, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4905, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.542045 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4907: 32150 keys, 33525504 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.542148 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000881542108, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4907, "file_size": 33525504, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126574819, "largest_seqno": 126656110, "table_properties": {"data_size": 32697472, "index_size": 827200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32697450, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4907, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.611826 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4909: 32053 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.611939 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000881611888, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4909, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126656111, "largest_seqno": 126758096, "table_properties": {"data_size": 32598208, "index_size": 925440, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32598191, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4909, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.655746 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004904.sst): fsize = 521.501 M, time(ms): open: 0.059, seek: 0.024, finish: 0.073, meta: 0.017, Flush: 0.021, sync: 357.879, close: 0.010, all: 358.083 2024/11/19-15:21:21.656867 139850967680768 [flush_job.cc:1083] [default] [JOB 159] Flush lasted 360086 microseconds, and 317162 cpu microseconds. 2024/11/19-15:21:21.664045 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4910: 32053 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.664093 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000881664072, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4910, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126758097, "largest_seqno": 126810536, "table_properties": {"data_size": 32597952, "index_size": 925696, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32597949, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4910, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.678642 139850967680768 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 15785682 2024/11/19-15:21:21.678728 139850967680768 (Original Log Time 2024/11/19-15:21:21.655803) [flush_job.cc:1029] [default] [JOB 159] Level-0 flush table #4904: 546833178 bytes OK (needs compaction) 2024/11/19-15:21:21.678734 139850967680768 (Original Log Time 2024/11/19-15:21:21.656884) [memtable_list.cc:560] [default] Level-0 commit table #4904 started 2024/11/19-15:21:21.678737 139850967680768 (Original Log Time 2024/11/19-15:21:21.678610) [memtable_list.cc:759] [default] Level-0 commit table #4904: memtable #1 done 2024/11/19-15:21:21.678740 139850967680768 (Original Log Time 2024/11/19-15:21:21.678635) EVENT_LOG_v1 {"time_micros": 1732000881678630, "job": 159, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [7, 27, 61, 22, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:21:21.678743 139850967680768 (Original Log Time 2024/11/19-15:21:21.678702) [db_impl_compaction_flush.cc:310] [default] Level summary: files[7 27 61 22 0 0 0] max score 1.51 (41 files need compaction) 2024/11/19-15:21:21.681235 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4908: 231492 keys, 33324931 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.681279 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000881681258, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4908, "file_size": 33324931, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 129015154, "largest_seqno": 129572893, "table_properties": {"data_size": 27026880, "index_size": 6297216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481776, "raw_average_key_size": 28, "raw_value_size": 27026880, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231492, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4908, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.694735 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4911: 32061 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.694785 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000881694757, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4911, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126771056, "largest_seqno": 126829128, "table_properties": {"data_size": 32609280, "index_size": 914880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641220, "raw_average_key_size": 20, "raw_value_size": 32609237, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32061, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4911, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.725299 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4913: 32118 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.725345 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000881725324, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4913, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126829129, "largest_seqno": 126904283, "table_properties": {"data_size": 32663744, "index_size": 860608, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642360, "raw_average_key_size": 20, "raw_value_size": 32663719, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32118, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4913, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.737347 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-02 2024/11/19-15:21:21.737444 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00146/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:21:21.837652 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4914: 32058 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.837712 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000881837690, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4914, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126904284, "largest_seqno": 127010109, "table_properties": {"data_size": 32602752, "index_size": 920960, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602691, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4914, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.871187 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4912: 231480 keys, 33324931 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.871253 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000881871224, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4912, "file_size": 33324931, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 129343352, "largest_seqno": 129983288, "table_properties": {"data_size": 27026624, "index_size": 6297472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481440, "raw_average_key_size": 28, "raw_value_size": 27026596, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231480, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4912, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.871788 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4915: 32058 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.871836 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000881871816, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4915, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127010110, "largest_seqno": 127639721, "table_properties": {"data_size": 32603328, "index_size": 920384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603280, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4915, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.903346 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4917: 32151 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.903396 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000881903376, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4917, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127639722, "largest_seqno": 127748480, "table_properties": {"data_size": 32696768, "index_size": 826880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32696755, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4917, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.953097 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4918: 32059 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.953152 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000881953130, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4918, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127748481, "largest_seqno": 127810549, "table_properties": {"data_size": 32602432, "index_size": 921664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32602372, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4918, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:21.964176 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4916: 179219 keys, 25838595 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:21.964225 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000881964204, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4916, "file_size": 25838595, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 129983289, "largest_seqno": 130202139, "table_properties": {"data_size": 20931712, "index_size": 4906048, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 5018132, "raw_average_key_size": 28, "raw_value_size": 20931694, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 179219, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4916, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.025763 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-00 2024/11/19-15:21:22.025855 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00155/att-00/compact.done) = No such file or directory: 11.022 sec on http probe failed: 4, retry 2024/11/19-15:21:22.045308 139849738749696 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4920: 1929 keys, 2100284 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.045383 139849738749696 EVENT_LOG_v1 {"time_micros": 1732000882045351, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4920, "file_size": 2100284, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 117781053, "largest_seqno": 117782981, "table_properties": {"data_size": 1962048, "index_size": 137408, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 38580, "raw_average_key_size": 20, "raw_value_size": 1961997, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 1929, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4920, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.045401 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4919: 32057 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.045450 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000882045429, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4919, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127757513, "largest_seqno": 127817521, "table_properties": {"data_size": 32604032, "index_size": 920192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32604029, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000881, "oldest_key_time": 0, "file_creation_time": 1732000881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4919, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.076952 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4921: 32149 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.077007 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000882076984, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4921, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127802604, "largest_seqno": 127883936, "table_properties": {"data_size": 32697984, "index_size": 826368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642980, "raw_average_key_size": 20, "raw_value_size": 32697959, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32149, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000882, "oldest_key_time": 0, "file_creation_time": 1732000882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4921, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.124191 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4922: 32060 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.124249 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000882124226, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4922, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127883937, "largest_seqno": 127962730, "table_properties": {"data_size": 32602944, "index_size": 921344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32602917, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000882, "oldest_key_time": 0, "file_creation_time": 1732000882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4922, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.186106 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-02 2024/11/19-15:21:22.186181 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00152/att-02/compact.done) = No such file or directory: 2.881 sec on http probe failed: 2, retry 2024/11/19-15:21:22.225396 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4923: 32058 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.225442 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000882225420, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4923, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127962731, "largest_seqno": 128636736, "table_properties": {"data_size": 32603328, "index_size": 920896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603308, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000882, "oldest_key_time": 0, "file_creation_time": 1732000882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4923, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.256964 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4924: 32141 keys, 33524800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.257015 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000882256995, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4924, "file_size": 33524800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128636737, "largest_seqno": 128738788, "table_properties": {"data_size": 32686016, "index_size": 837952, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642820, "raw_average_key_size": 20, "raw_value_size": 32685986, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32141, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000882, "oldest_key_time": 0, "file_creation_time": 1732000882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4924, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.288380 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4925: 32057 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.288431 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000882288409, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4925, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128738789, "largest_seqno": 128801757, "table_properties": {"data_size": 32603328, "index_size": 920896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32603290, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000882, "oldest_key_time": 0, "file_creation_time": 1732000882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4925, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.302529 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-00 2024/11/19-15:21:22.302621 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00156/att-00/compact.done) = No such file or directory: 7.763 sec on http probe failed: 3, retry 2024/11/19-15:21:22.400678 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4926: 32057 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.400731 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000882400709, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4926, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128746200, "largest_seqno": 128826210, "table_properties": {"data_size": 32601792, "index_size": 922304, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601731, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000882, "oldest_key_time": 0, "file_creation_time": 1732000882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4926, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.432022 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4927: 32151 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.432071 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000882432050, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4927, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128773810, "largest_seqno": 128844974, "table_properties": {"data_size": 32697088, "index_size": 826624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32697065, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000882, "oldest_key_time": 0, "file_creation_time": 1732000882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4927, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.462976 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4928: 32056 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.463027 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000882463006, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4928, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128844975, "largest_seqno": 128917240, "table_properties": {"data_size": 32599360, "index_size": 924288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32599330, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000882, "oldest_key_time": 0, "file_creation_time": 1732000882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4928, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.594848 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4929: 32059 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.594903 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000882594881, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4929, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128917241, "largest_seqno": 129627867, "table_properties": {"data_size": 32601216, "index_size": 923008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32601158, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000882, "oldest_key_time": 0, "file_creation_time": 1732000882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4929, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.626217 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4930: 32100 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.626269 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000882626248, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4930, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 129627868, "largest_seqno": 129697003, "table_properties": {"data_size": 32644928, "index_size": 879424, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642000, "raw_average_key_size": 20, "raw_value_size": 32644925, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32100, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000882, "oldest_key_time": 0, "file_creation_time": 1732000882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4930, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.657609 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4931: 32053 keys, 33524800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.657662 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000882657641, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4931, "file_size": 33524800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 129697004, "largest_seqno": 129773071, "table_properties": {"data_size": 32600832, "index_size": 923136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32600783, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000882, "oldest_key_time": 0, "file_creation_time": 1732000882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4931, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.689055 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4932: 32056 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.689106 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000882689085, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4932, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 129755044, "largest_seqno": 129815052, "table_properties": {"data_size": 32601280, "index_size": 922816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32601222, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000882, "oldest_key_time": 0, "file_creation_time": 1732000882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4932, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.821668 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4933: 32136 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.821723 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000882821701, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4933, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 129775130, "largest_seqno": 129836278, "table_properties": {"data_size": 32682240, "index_size": 841920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642720, "raw_average_key_size": 20, "raw_value_size": 32682181, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32136, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000882, "oldest_key_time": 0, "file_creation_time": 1732000882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4933, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.853085 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4934: 32057 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.853136 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000882853115, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4934, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 129836279, "largest_seqno": 129908180, "table_properties": {"data_size": 32602368, "index_size": 921856, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602351, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000882, "oldest_key_time": 0, "file_creation_time": 1732000882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4934, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.878335 139849728263936 [compaction_job.cc:1948] [default] [JOB 158] Generated table #4935: 25725 keys, 26900480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.878386 139849728263936 EVENT_LOG_v1 {"time_micros": 1732000882878365, "cf_name": "default", "job": 158, "event": "table_file_creation", "file_number": 4935, "file_size": 26900480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 129908181, "largest_seqno": 129964546, "table_properties": {"data_size": 26161280, "index_size": 738368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 514500, "raw_average_key_size": 20, "raw_value_size": 26161261, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 25725, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732000882, "oldest_key_time": 0, "file_creation_time": 1732000882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4935, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:21:22.878483 139851148035840 [compaction_job.cc:703] job-00158: subcompact[0], size: 0.353680 G, files: /storage/mytopling/.rocksdb [004816,004819,004820,004822,004824,004827,004829,004833,004834,004836,004838,004840,004843,004844] 2024/11/19-15:21:22.878509 139851148035840 [compaction_job.cc:703] job-00158: subcompact[1], size: 0.776795 G, files: /storage/mytopling/.rocksdb [004817,004826,004831,004841,004846,004849,004853,004857,004861,004865,004869,004872,004874,004876,004880,004882,004886,004890,004893,004896,004899,004905,004908,004912,004916,004920] 2024/11/19-15:21:22.878538 139851148035840 [compaction_job.cc:703] job-00158: subcompact[2], size: 2.608310 G, files: /storage/mytopling/.rocksdb [004818,004821,004823,004825,004828,004830,004832,004835,004837,004839,004842,004845,004847,004848,004850,004851,004852,004854,004855,004856,004858,004859,004860,004862,004863,004864,004866,004867,004868,004870,004871,004873,004875,004877,004878,004879,004881,004883,004884,004885,004887,004888,004889,004891,004892,004894,004895,004897,004898,004900,004901,004902,004906,004907,004909,004910,004911,004913,004914,004915,004917,004918,004919,004921,004922,004923,004924,004925,004926,004927,004928,004929,004930,004931,004932,004933,004934,004935] 2024/11/19-15:21:22.997545 139851148035840 (Original Log Time 2024/11/19-15:21:18.300177) EVENT_LOG_v1 {"time_micros": 1732000878300169, "job": 158, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [4799, 4786, 4777, 4768, 4759, 4746], "files_L1": [4715, 4716, 4717, 4718, 4719, 4720, 4721, 4722, 4723, 4724, 4725, 4726, 4727, 4728, 4729, 4730, 4731, 4732, 4733, 4734, 4735, 4736, 4737, 4738, 4739, 4740, 4741], "score": 3.62083, "input_data_size": 4142649244, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:22.997561 139851148035840 (Original Log Time 2024/11/19-15:21:22.996654) [compaction_job.cc:2022] [default] [JOB 158] Compacted 6@0 + 27@1 files to L1 => 3738785019 bytes 2024/11/19-15:21:22.997565 139851148035840 (Original Log Time 2024/11/19-15:21:22.997196) [compaction_job.cc:1224] [default] compacted to: files[1 118 61 22 0 0 0] max score 6.97 (159 files need compaction), MB/sec: 904.8 rd, 816.6 wr, level 1, files in(6, 27) out(118 +0 blob) MB in(3095.8, 855.0 +0.0 blob) out(3565.6 +0.0 blob), read-write-amplify(2.4) write-amplify(1.2) OK, records in: 9195210, records dropped: 0 output_compression: Snappy 2024/11/19-15:21:22.997570 139851148035840 (Original Log Time 2024/11/19-15:21:22.997255) EVENT_LOG_v1 {"time_micros": 1732000882997210, "cf": "default", "job": 158, "event": "compaction_finished", "compaction_time_micros": 4578364, "compaction_time_cpu_micros": 6051441, "output_level": 1, "num_output_files": 118, "total_output_size": 3738785019, "num_input_records": 9195210, "num_output_records": 9195210, "num_subcompactions": 3, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 118, 61, 22, 0, 0, 0]} 2024/11/19-15:21:22.997710 139851148035840 [compaction_job.cc:2437] [__system__] [JOB 160] Compacting 1@0 + 1@1 files to L1, score 0.25, subcompactions 3 : 1 2024/11/19-15:21:22.997719 139851148035840 [compaction_job.cc:2445] [__system__]: Compaction start summary: Base version 155 Base level 0, inputs: [4813(7770B)], [4337(5672B)] 2024/11/19-15:21:22.998129 139851148035840 [compaction_job.cc:1948] [__system__] [JOB 160] Generated table #4936: 34 keys, 5800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:22.998164 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000882998146, "cf_name": "__system__", "job": 160, "event": "table_file_creation", "file_number": 4936, "file_size": 5800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 1344, "index_size": 3648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 803, "raw_average_key_size": 23, "raw_value_size": 1281, "raw_average_value_size": 37, "num_data_blocks": 1, "num_entries": 34, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "__system__", "column_family_id": 1, "comparator": "leveldb.BytewiseComparator", "merge_operator": "Rdb_system_merge_op", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732000882, "oldest_key_time": 0, "file_creation_time": 1732000882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4936, "seqno_to_time_mapping": "N/A"}} 2024/11/19-15:21:22.998189 139851148035840 [compaction_job.cc:703] job-00160: subcompact[0], size: 0.000006 G, files: /storage/mytopling/.rocksdb [004936] 2024/11/19-15:21:22.998434 139851148035840 (Original Log Time 2024/11/19-15:21:22.997733) EVENT_LOG_v1 {"time_micros": 1732000882997726, "job": 160, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L0": [4813], "files_L1": [4337], "score": 0.25, "input_data_size": 13442, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:22.998441 139851148035840 (Original Log Time 2024/11/19-15:21:22.998319) [compaction_job.cc:2022] [__system__] [JOB 160] Compacted 1@0 + 1@1 files to L1 => 5800 bytes 2024/11/19-15:21:22.998444 139851148035840 (Original Log Time 2024/11/19-15:21:22.998358) [compaction_job.cc:1224] [__system__] compacted to: files[0 1 0 0 0 0 0] max score 0.00, MB/sec: 29.2 rd, 12.6 wr, level 1, files in(1, 1) out(1 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(2.5) write-amplify(0.7) OK, records in: 108, records dropped: 74 output_compression: Snappy 2024/11/19-15:21:22.998451 139851148035840 (Original Log Time 2024/11/19-15:21:22.998374) EVENT_LOG_v1 {"time_micros": 1732000882998366, "cf": "__system__", "job": 160, "event": "compaction_finished", "compaction_time_micros": 461, "compaction_time_cpu_micros": 341, "output_level": 1, "num_output_files": 1, "total_output_size": 5800, "num_input_records": 108, "num_output_records": 34, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]} 2024/11/19-15:21:22.998615 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004337.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:22.998628 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000882998626, "job": 160, "event": "table_file_deletion", "file_number": 4337} 2024/11/19-15:21:22.998736 139851148035840 [compaction_job.cc:2437] [default] [JOB 161] Compacting 2@1 files to L2, score 6.97, subcompactions 1 : 1 2024/11/19-15:21:22.998754 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 158 Base level 1, inputs: [4816(6067B) 4819(28KB)] 2024/11/19-15:21:23.293483 139851148035840 [dcompact_etcd.cc:1163] job-00161/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.294 sec, input = 25.090 KiB 2024/11/19-15:21:23.367771 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-00 2024/11/19-15:21:23.367841 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00161/att-00/compact.done) = No such file or directory: 0.065 sec on http probe failed: 1, retry 2024/11/19-15:21:23.376526 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-00 2024/11/19-15:21:23.376577 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00161/att-00/compact.done) = No such file or directory: 0.075 sec on http probe failed: 2, retry 2024/11/19-15:21:23.386385 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-00 2024/11/19-15:21:23.386434 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00161/att-00/compact.done) = No such file or directory: 0.084 sec on http probe failed: 3, retry 2024/11/19-15:21:23.394803 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-00 2024/11/19-15:21:23.394857 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00161/att-00/compact.done) = No such file or directory: 0.094 sec on http probe failed: 4, retry 2024/11/19-15:21:23.403367 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-00 2024/11/19-15:21:23.403416 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00161/att-00/compact.done) = No such file or directory: 0.102 sec on http probe failed: 5, giveup 2024/11/19-15:21:23.403433 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00161/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.404 (submit 294.223 ms wait 0.102 s), input 25.090 KiB, send shutdown 2024/11/19-15:21:23.410870 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-00 2024/11/19-15:21:23.411094 139851148035840 [dcompact_etcd.cc:1773] job-00161/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:23.467774 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-02 2024/11/19-15:21:23.467843 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00152/att-02/compact.done) = No such file or directory: 4.166 sec on http probe failed: 3, retry 2024/11/19-15:21:23.714832 139851148035840 [dcompact_etcd.cc:1163] job-00161/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.304 sec, input = 25.090 KiB 2024/11/19-15:21:23.742881 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-02 2024/11/19-15:21:23.742957 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00146/att-02/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:21:23.785551 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-01 2024/11/19-15:21:23.785621 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00161/att-01/compact.done) = No such file or directory: 0.063 sec on http probe failed: 1, retry 2024/11/19-15:21:23.792691 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-01 2024/11/19-15:21:23.792758 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00161/att-01/compact.done) = No such file or directory: 0.071 sec on http probe failed: 2, retry 2024/11/19-15:21:23.801972 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-01 2024/11/19-15:21:23.802021 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00161/att-01/compact.done) = No such file or directory: 0.078 sec on http probe failed: 3, retry 2024/11/19-15:21:23.806807 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-01 2024/11/19-15:21:23.806853 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00161/att-01/compact.done) = No such file or directory: 0.088 sec on http probe failed: 4, retry 2024/11/19-15:21:23.812518 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-01 2024/11/19-15:21:23.812561 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00161/att-01/compact.done) = No such file or directory: 0.093 sec on http probe failed: 5, giveup 2024/11/19-15:21:23.812575 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00161/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.401 (submit 303.614 ms wait 0.093 s), input 25.090 KiB, send shutdown 2024/11/19-15:21:23.816864 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-01 2024/11/19-15:21:23.817152 139851148035840 [dcompact_etcd.cc:1773] job-00161/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:24.034040 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-00 2024/11/19-15:21:24.034117 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00155/att-00/compact.done) = No such file or directory: 13.031 sec on http probe failed: 5, giveup 2024/11/19-15:21:24.034135 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00155/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.634 sec, real 13.348 (submit 309.188 ms wait 13.031 s), input 255.634 MiB, send shutdown 2024/11/19-15:21:24.042699 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-00 2024/11/19-15:21:24.081788 139851148035840 [dcompact_etcd.cc:1163] job-00161/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.265 sec, input = 25.090 KiB 2024/11/19-15:21:24.161008 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-02 2024/11/19-15:21:24.161088 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00161/att-02/compact.done) = No such file or directory: 0.070 sec on http probe failed: 1, retry 2024/11/19-15:21:24.169285 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-02 2024/11/19-15:21:24.169347 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00161/att-02/compact.done) = No such file or directory: 0.080 sec on http probe failed: 2, retry 2024/11/19-15:21:24.177327 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-02 2024/11/19-15:21:24.177384 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00161/att-02/compact.done) = No such file or directory: 0.088 sec on http probe failed: 3, retry 2024/11/19-15:21:24.183893 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-02 2024/11/19-15:21:24.183953 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00161/att-02/compact.done) = No such file or directory: 0.096 sec on http probe failed: 4, retry 2024/11/19-15:21:24.190948 139851158521600 [dcompact_etcd.cc:1773] job-00155/att-00: CleanFiles 0.148 sec : 0.000 2024/11/19-15:21:24.194882 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-02 2024/11/19-15:21:24.194939 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00161/att-02/compact.done) = No such file or directory: 0.103 sec on http probe failed: 5, giveup 2024/11/19-15:21:24.194955 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00161/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.378 (submit 264.513 ms wait 0.103 s), input 25.090 KiB, send shutdown 2024/11/19-15:21:24.199890 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00161/att-02 2024/11/19-15:21:24.307782 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-00 2024/11/19-15:21:24.307855 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00156/att-00/compact.done) = No such file or directory: 9.773 sec on http probe failed: 4, retry 2024/11/19-15:21:24.470602 139851158521600 [dcompact_etcd.cc:1163] job-00155/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 255.634 MiB 2024/11/19-15:21:24.477328 139851148035840 [dcompact_etcd.cc:1773] job-00161/att-02: CleanFiles 0.277 sec : 0.000 2024/11/19-15:21:24.479153 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00161) = success 2024/11/19-15:21:24.479163 139851148035840 [dcompact_etcd.cc:1773] job-00161/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:21:24.479184 139851148035840 (Original Log Time 2024/11/19-15:21:22.998765) EVENT_LOG_v1 {"time_micros": 1732000882998760, "job": 161, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4816, 4819], "score": 6.97137, "input_data_size": 34987, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:24.479191 139851148035840 [compaction_job.cc:2437] [default] [JOB 161] Compacting 2@1 files to L2, score 6.97, subcompactions 1 : 1 2024/11/19-15:21:24.479198 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 158 Base level 1, inputs: [4816(6067B) 4819(28KB)] 2024/11/19-15:21:24.479453 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004937.sst) /storage/mytopling/..//tmp/Topling-82038-hwRqYA 2024/11/19-15:21:24.479587 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004937.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:21:24.479665 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004937.sst): index pass time = 0.00's, 1.044'MB/sec index type = UintIndex_AllOne usrkeys = 16 min-keylen = 4 max-keylen = 4 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 10.00 2024/11/19-15:21:24.479732 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004937.sst): first pass time = 0.00's, 7.920'MB/sec 2024/11/19-15:21:24.479760 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004937.sst): old prealloc_size = 38485, real_size = 2480 2024/11/19-15:21:24.479775 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004937.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 16 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 10.00 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 117.25 avg-zval = 132.00 2024/11/19-15:21:24.480005 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004937.sst): second pass time = 0.00's,13496.403'MB/sec, value only(88.0% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 15.398'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 11.538'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.153'MB/sec write SST data time = 0.00's, 18.831'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 13496.403'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 16 avg-key = 16.00 avg-zkey = 12.00 avg-val = 117.25 avg-zval = 132.00 usrkeys = 16 avg-key = 8.00 avg-zkey = 12.00 avg-val = 124.25 avg-zval = 132.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 16 bytes = 64 } seq expand size = 112 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.6667 value = 0.8883 dict = 0.00 all = 0.6052 } Zip/UnZip{ index = 1.5000 value = 1.1258 dict = 0.00 all = 1.6524 } ---------------------------- total value len = 7.746081 GB avg = 0.281 KB (by entry num) total key len = 0.699146 GB avg = 0.025 KB total ukey len = 0.478920 GB avg = 0.017 KB total ukey num = 0.027528176 Billion total entry num = 0.027528176 Billion write speed all = 50.415701221 MB/sec (with seq num) write speed all = 49.101015623 MB/sec (without seq num) 2024/11/19-15:21:24.480188 139851148035840 [compaction_job.cc:1948] [default] [JOB 161] Generated table #4937: 16 keys, 3523 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:24.480223 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000884480203, "cf_name": "default", "job": 161, "event": "table_file_creation", "file_number": 4937, "file_size": 3523, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 122618538, "largest_seqno": 129745041, "table_properties": {"data_size": 2112, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 256, "raw_average_key_size": 16, "raw_value_size": 1876, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 16, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000884, "oldest_key_time": 0, "file_creation_time": 1732000884, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4937, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:24.480386 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004938.sst) /storage/mytopling/..//tmp/Topling-82038-A9PEQD 2024/11/19-15:21:24.480527 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004938.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:21:24.480662 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004938.sst): first pass time = 0.00's, 87.469'MB/sec 2024/11/19-15:21:24.480672 139834035275520 [fixed_len_key_index.cc:363] FixedLenKeyIndex build_cache Finish: num_keys = 167, fixlen = 8, pref_len = 4, cplen = 3, time = 0.000 sec 2024/11/19-15:21:24.480735 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004938.sst): index pass time = 0.00's, 6.807'MB/sec index type = FixedLenKeyIndex usrkeys = 167 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 60.17 2024/11/19-15:21:24.480775 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004938.sst): old prealloc_size = 38485, real_size = 31008 2024/11/19-15:21:24.480790 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004938.sst): index type = FixedLenKeyIndex , store type = PlainBlobStore usrkeys = 167 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 60.17 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 121.08 avg-zval = 124.26 2024/11/19-15:21:24.480996 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004938.sst): second pass time = 0.00's,342711.864'MB/sec, value only(85.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 152.579'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 642.332'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 1.352'MB/sec write SST data time = 0.00's, 194.100'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 342711.864'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 167 avg-key = 20.00 avg-zkey = 60.17 avg-val = 121.08 avg-zval = 124.26 usrkeys = 167 avg-key = 12.00 avg-zkey = 60.17 avg-val = 128.08 avg-zval = 124.26 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 167 bytes = 496 } seq expand size = 1169 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.1994 value = 0.9744 dict = 0.00 all = 0.7257 } Zip/UnZip{ index = 5.0140 value = 1.0263 dict = 0.00 all = 1.3779 } ---------------------------- total value len = 7.746103 GB avg = 0.281 KB (by entry num) total key len = 0.699149 GB avg = 0.025 KB total ukey len = 0.478922 GB avg = 0.017 KB total ukey num = 0.027528343 Billion total entry num = 0.027528343 Billion write speed all = 50.415549749 MB/sec (with seq num) write speed all = 49.100863975 MB/sec (without seq num) 2024/11/19-15:21:24.481306 139851148035840 [compaction_job.cc:1948] [default] [JOB 161] Generated table #4938: 167 keys, 32464 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:24.481335 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000884481318, "cf_name": "default", "job": 161, "event": "table_file_creation", "file_number": 4938, "file_size": 32464, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121930472, "largest_seqno": 130124230, "table_properties": {"data_size": 20752, "index_size": 10048, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3340, "raw_average_key_size": 20, "raw_value_size": 20220, "raw_average_value_size": 121, "num_data_blocks": 1, "num_entries": 167, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-FixedLenKeyIndex-PlainBlobStore", "creation_time": 1732000884, "oldest_key_time": 0, "file_creation_time": 1732000884, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4938, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:24.481424 139851148035840 [compaction_job.cc:703] job-00161: subcompact[0], size: 0.000036 G, files: /storage/mytopling/.rocksdb [004937,004938] 2024/11/19-15:21:24.482101 139851148035840 (Original Log Time 2024/11/19-15:21:24.479230) EVENT_LOG_v1 {"time_micros": 1732000884479211, "job": 161, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4816, 4819], "score": 6.97137, "input_data_size": 34987, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:24.482110 139851148035840 (Original Log Time 2024/11/19-15:21:24.481681) [compaction_job.cc:2022] [default] [JOB 161] Compacted 2@1 files to L2 => 35987 bytes 2024/11/19-15:21:24.482113 139851148035840 (Original Log Time 2024/11/19-15:21:24.482002) [compaction_job.cc:1224] [default] compacted to: files[1 116 63 22 0 0 0] max score 6.97 (159 files need compaction), MB/sec: 15.9 rd, 16.4 wr, level 2, files in(2, 0) out(2 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 183, records dropped: 0 output_compression: Snappy 2024/11/19-15:21:24.482124 139851148035840 (Original Log Time 2024/11/19-15:21:24.482019) EVENT_LOG_v1 {"time_micros": 1732000884482010, "cf": "default", "job": 161, "event": "compaction_finished", "compaction_time_micros": 2199, "compaction_time_cpu_micros": 1777, "output_level": 2, "num_output_files": 2, "total_output_size": 35987, "num_input_records": 183, "num_output_records": 183, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 116, 63, 22, 0, 0, 0]} 2024/11/19-15:21:24.482190 139851148035840 [compaction_job.cc:2437] [default] [JOB 162] Compacting 4@1 files to L2, score 6.97, subcompactions 1 : 1 2024/11/19-15:21:24.482198 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 160 Base level 1, inputs: [4820(31MB) 4822(31MB) 4824(31MB) 4827(31MB)] 2024/11/19-15:21:24.750285 139851148035840 [dcompact_etcd.cc:1163] job-00162/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.268 sec, input = 127.238 MiB 2024/11/19-15:21:24.752930 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-02 2024/11/19-15:21:24.752991 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00152/att-02/compact.done) = No such file or directory: 5.448 sec on http probe failed: 4, retry 2024/11/19-15:21:25.733332 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-01 2024/11/19-15:21:25.733421 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00153/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:21:25.751546 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-02 2024/11/19-15:21:25.751622 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00146/att-02/compact.done) = No such file or directory: 9.015 sec on http probe failed: 3, retry 2024/11/19-15:21:25.836224 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-01 2024/11/19-15:21:25.836334 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00154/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:21:26.038200 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-02 2024/11/19-15:21:26.038277 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00152/att-02/compact.done) = No such file or directory: 6.733 sec on http probe failed: 5, giveup 2024/11/19-15:21:26.038295 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00152/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 63.819 sec, real 6.996 (submit 254.390 ms wait 6.733 s), input 63.819 MiB, send shutdown 2024/11/19-15:21:26.045178 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00152/att-02 2024/11/19-15:21:26.045402 139850978166528 [dcompact_etcd.cc:1773] job-00152/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:26.047338 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00152) = success 2024/11/19-15:21:26.047347 139850978166528 [dcompact_etcd.cc:1773] job-00152/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:21:26.047366 139850978166528 (Original Log Time 2024/11/19-15:21:04.942897) EVENT_LOG_v1 {"time_micros": 1732000864942891, "job": 152, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4599], "score": 1.94829, "input_data_size": 55363416, "oldest_snapshot_seqno": 130782877} 2024/11/19-15:21:26.047372 139850978166528 [compaction_job.cc:2437] [default] [JOB 152] Compacting 1@2 files to L3, score 1.95, subcompactions 1 : 1 2024/11/19-15:21:26.047380 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 150 Base level 2, inputs: [4599(52MB)] 2024/11/19-15:21:26.047634 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004939.sst) /storage/mytopling/..//tmp/Topling-82038-YT8GjK 2024/11/19-15:21:26.230555 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004939.sst): first pass time = 0.18's, 365.860'MB/sec 2024/11/19-15:21:26.230557 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004939.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:21:26.230593 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004939.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.006 GB, dictZip workingMem = 0.0033 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:21:26.265302 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004939.sst): index pass time = 0.03's, 213.106'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462275 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:21:26.314594 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-00 2024/11/19-15:21:26.314684 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00156/att-00/compact.done) = No such file or directory: 11.778 sec on http probe failed: 5, giveup 2024/11/19-15:21:26.314704 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00156/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 149.879 sec, real 12.088 (submit 302.312 ms wait 11.778 s), input 149.879 MiB, send shutdown 2024/11/19-15:21:26.320068 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-00 2024/11/19-15:21:26.320283 139851127064320 [dcompact_etcd.cc:1773] job-00156/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:26.602443 139851127064320 [dcompact_etcd.cc:1163] job-00156/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.282 sec, input = 149.879 MiB 2024/11/19-15:21:26.902195 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004939.sst): old prealloc_size = 60899757, real_size = 18305712 2024/11/19-15:21:26.902443 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004939.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 462275 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0173 GB avg-val = 116.76 avg-zval = 37.44 2024/11/19-15:21:26.902520 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004939.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.672 sec, Key+Value = 0.067 GB 2024/11/19-15:21:27.162616 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004939.sst): second pass time = 0.67's, 80.371'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.26's, 71.963'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3748.547'MB/sec (index lex order gen) rebuild zvType time = 0.21's, 0.542'MB/sec write SST data time = 0.05's, 399.124'MB/sec dict compress time = 0.00's, 1244.274'MB/sec z-dict build time = 0.00's, sample length = 0.544'MB, throughput = 7441.330'MB/sec zip my value time = 0.67's, unzip length = 0.054'GB zip my value throughput = 80.371'MB/sec zip pipeline throughput = 80.371'MB/sec entries = 462275 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 37.44 usrkeys = 462275 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 37.44 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.54 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0173 GB dict = 0.54 MB all = 0.0187 GB } UnZip/Zip{ index = 10.5024 value = 3.1182 dict = 1.00 all = 3.5711 } Zip/UnZip{ index = 0.0952 value = 0.3207 dict = 1.00 all = 0.2800 } ---------------------------- total value len = 7.800078 GB avg = 0.279 KB (by entry num) total key len = 0.712093 GB avg = 0.025 KB total ukey len = 0.488168 GB avg = 0.017 KB total ukey num = 0.027990618 Billion total entry num = 0.027990618 Billion write speed all = 50.014385825 MB/sec (with seq num) write speed all = 48.698685137 MB/sec (without seq num) 2024/11/19-15:21:27.163535 139850978166528 [compaction_job.cc:1948] [default] [JOB 152] Generated table #4939: 462275 keys, 18739195 bytes, temperature: kUnknown 2024/11/19-15:21:27.163587 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000887163563, "cf_name": "default", "job": 152, "event": "table_file_creation", "file_number": 4939, "file_size": 18739195, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 17309664, "index_size": 880320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12943700, "raw_average_key_size": 28, "raw_value_size": 53974994, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462275, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000886, "oldest_key_time": 0, "file_creation_time": 1732000886, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4939, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:27.177777 139850978166528 [compaction_job.cc:703] job-00152: subcompact[0], size: 0.018739 G, files: /storage/mytopling/.rocksdb [004939] 2024/11/19-15:21:27.178865 139850978166528 (Original Log Time 2024/11/19-15:21:26.047409) EVENT_LOG_v1 {"time_micros": 1732000886047391, "job": 152, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4599], "score": 1.94829, "input_data_size": 55363416, "oldest_snapshot_seqno": 130782877} 2024/11/19-15:21:27.178875 139850978166528 (Original Log Time 2024/11/19-15:21:27.178423) [compaction_job.cc:2022] [default] [JOB 152] Compacted 1@2 files to L3 => 18739195 bytes 2024/11/19-15:21:27.178878 139850978166528 (Original Log Time 2024/11/19-15:21:27.178753) [compaction_job.cc:1224] [default] compacted to: files[1 116 62 23 0 0 0] max score 6.72 (155 files need compaction), MB/sec: 49.0 rd, 16.6 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(52.8, 0.0 +0.0 blob) out(17.9 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 462275, records dropped: 0 output_compression: Snappy 2024/11/19-15:21:27.178893 139850978166528 (Original Log Time 2024/11/19-15:21:27.178777) EVENT_LOG_v1 {"time_micros": 1732000887178766, "cf": "default", "job": 152, "event": "compaction_finished", "compaction_time_micros": 1130382, "compaction_time_cpu_micros": 903746, "output_level": 3, "num_output_files": 1, "total_output_size": 18739195, "num_input_records": 462275, "num_output_records": 462275, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 116, 62, 23, 0, 0, 0]} 2024/11/19-15:21:27.178986 139850978166528 [compaction_job.cc:2437] [default] [JOB 163] Compacting 4@1 files to L2, score 6.72, subcompactions 1 : 1 2024/11/19-15:21:27.178994 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 161 Base level 1, inputs: [4829(31MB) 4833(31MB) 4834(31MB) 4836(31MB)] 2024/11/19-15:21:27.493822 139850978166528 [dcompact_etcd.cc:1163] job-00163/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.314 sec, input = 127.240 MiB 2024/11/19-15:21:27.743584 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-01 2024/11/19-15:21:27.743677 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00153/att-01/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:21:27.759345 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-02 2024/11/19-15:21:27.759420 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00146/att-02/compact.done) = No such file or directory: 11.024 sec on http probe failed: 4, retry 2024/11/19-15:21:27.845243 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-01 2024/11/19-15:21:27.845350 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00154/att-01/compact.done) = No such file or directory: 7.008 sec on http probe failed: 2, retry 2024/11/19-15:21:27.939305 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-00 2024/11/19-15:21:27.939424 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00162/att-00/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:21:29.478956 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-01 2024/11/19-15:21:29.479041 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00155/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:21:29.751406 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-01 2024/11/19-15:21:29.751489 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00153/att-01/compact.done) = No such file or directory: 9.019 sec on http probe failed: 3, retry 2024/11/19-15:21:29.764073 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-02 2024/11/19-15:21:29.764150 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00146/att-02/compact.done) = No such file or directory: 13.032 sec on http probe failed: 5, giveup 2024/11/19-15:21:29.764174 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00146/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 254.479 sec, real 13.327 (submit 290.346 ms wait 13.032 s), input 254.479 MiB, send shutdown 2024/11/19-15:21:29.769589 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00146/att-02 2024/11/19-15:21:29.769845 139851137550080 [dcompact_etcd.cc:1773] job-00146/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:29.771779 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00146) = success 2024/11/19-15:21:29.771792 139851137550080 [dcompact_etcd.cc:1773] job-00146/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:21:29.771817 139851137550080 (Original Log Time 2024/11/19-15:20:49.791361) EVENT_LOG_v1 {"time_micros": 1732000849791353, "job": 146, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4763, 4764, 4765, 4766], "score": 1.85563, "input_data_size": 259250036, "oldest_snapshot_seqno": 129913905} 2024/11/19-15:21:29.771829 139851137550080 [compaction_job.cc:2437] [default] [JOB 146] Compacting 4@2 files to L3, score 1.86, subcompactions 1 : 1 2024/11/19-15:21:29.771841 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 144 Base level 2, inputs: [4763(62MB) 4764(61MB) 4765(62MB) 4766(61MB)] 2024/11/19-15:21:29.772184 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004940.sst) /storage/mytopling/..//tmp/Topling-82038-FA47dh 2024/11/19-15:21:29.853476 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-01 2024/11/19-15:21:29.853571 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00154/att-01/compact.done) = No such file or directory: 9.017 sec on http probe failed: 3, retry 2024/11/19-15:21:29.920892 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004940.sst): first pass time = 0.15's, 902.705'MB/sec 2024/11/19-15:21:29.920888 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004940.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0018 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:21:29.920935 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004940.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.002 GB, dictZip workingMem = 0.0081 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:21:29.931293 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004940.sst): index pass time = 0.01's, 232.309'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 200584 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 2024/11/19-15:21:29.947838 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-00 2024/11/19-15:21:29.947933 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00162/att-00/compact.done) = No such file or directory: 5.189 sec on http probe failed: 2, retry 2024/11/19-15:21:30.363358 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-01 2024/11/19-15:21:30.363464 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00156/att-01/compact.done) = No such file or directory: 3.747 sec on http probe failed: 1, retry 2024/11/19-15:21:30.689486 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-00 2024/11/19-15:21:30.689634 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00163/att-00/compact.done) = No such file or directory: 3.184 sec on http probe failed: 1, retry 2024/11/19-15:21:31.500089 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-01 2024/11/19-15:21:31.500188 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00155/att-01/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:21:31.768004 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-01 2024/11/19-15:21:31.768107 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00153/att-01/compact.done) = No such file or directory: 11.029 sec on http probe failed: 4, retry 2024/11/19-15:21:31.872987 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-01 2024/11/19-15:21:31.873084 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00154/att-01/compact.done) = No such file or directory: 11.025 sec on http probe failed: 4, retry 2024/11/19-15:21:31.965984 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-00 2024/11/19-15:21:31.966080 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00162/att-00/compact.done) = No such file or directory: 7.198 sec on http probe failed: 3, retry 2024/11/19-15:21:32.162979 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004940.sst): old prealloc_size = 147639500, real_size = 103792128 2024/11/19-15:21:32.163104 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004940.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 200584 prefix = 4 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 raw-val = 0.1294 GB zip-val = 0.1034 GB avg-val = 645.14 avg-zval = 515.51 2024/11/19-15:21:32.163153 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004940.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0005 GB, waited 2.242 sec, Key+Value = 0.134 GB 2024/11/19-15:21:32.364658 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004940.sst): second pass time = 2.24's, 57.718'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.20's, 521.118'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3205.605'MB/sec (index lex order gen) rebuild zvType time = 0.16's, 0.316'MB/sec write SST data time = 0.04's, 2462.682'MB/sec dict compress time = 0.00's, 2140.870'MB/sec z-dict build time = 0.00's, sample length = 1.346'MB, throughput = 29788.854'MB/sec zip my value time = 2.24's, unzip length = 0.129'GB zip my value throughput = 57.718'MB/sec zip pipeline throughput = 57.718'MB/sec entries = 200584 avg-key = 24.00 avg-zkey = 1.69 avg-val = 645.14 avg-zval = 515.51 usrkeys = 200584 avg-key = 16.00 avg-zkey = 1.69 avg-val = 645.14 avg-zval = 515.51 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0032 GB value = 0.1294 GB dict = 1.35 MB all = 0.1342 GB } __ZipSize{ index = 0.0003 GB value = 0.1034 GB dict = 1.35 MB all = 0.1051 GB } UnZip/Zip{ index = 9.4794 value = 1.2515 dict = 1.00 all = 1.2772 } Zip/UnZip{ index = 0.1055 value = 0.7991 dict = 1.00 all = 0.7830 } ---------------------------- total value len = 7.929482 GB avg = 0.281 KB (by entry num) total key len = 0.716907 GB avg = 0.025 KB total ukey len = 0.491377 GB avg = 0.017 KB total ukey num = 0.028191202 Billion total entry num = 0.028191202 Billion write speed all = 49.296248318 MB/sec (with seq num) write speed all = 48.010420873 MB/sec (without seq num) 2024/11/19-15:21:32.372616 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-01 2024/11/19-15:21:32.372717 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00156/att-01/compact.done) = No such file or directory: 5.761 sec on http probe failed: 2, retry 2024/11/19-15:21:32.383018 139851137550080 [compaction_job.cc:1948] [default] [JOB 146] Generated table #4940: 200584 keys, 105090681 bytes, temperature: kUnknown 2024/11/19-15:21:32.383069 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000892383045, "cf_name": "default", "job": 146, "event": "table_file_creation", "file_number": 4940, "file_size": 105090681, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 103403248, "index_size": 338560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4814016, "raw_average_key_size": 24, "raw_value_size": 129404260, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 200584, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000889, "oldest_key_time": 0, "file_creation_time": 1732000889, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4940, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:32.409441 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004941.sst) /storage/mytopling/..//tmp/Topling-82038-N16FwP 2024/11/19-15:21:32.559076 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004941.sst): first pass time = 0.15's, 886.402'MB/sec 2024/11/19-15:21:32.559086 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004941.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0018 GB, waited 0.000 sec, Key+Value = 0.133 GB 2024/11/19-15:21:32.559120 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004941.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.002 GB, dictZip workingMem = 0.0080 GB, waited 0.000 sec, Key+Value = 0.133 GB 2024/11/19-15:21:32.573711 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004941.sst): index pass time = 0.01's, 163.066'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 198262 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 2024/11/19-15:21:32.698572 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-00 2024/11/19-15:21:32.698666 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00163/att-00/compact.done) = No such file or directory: 5.196 sec on http probe failed: 2, retry 2024/11/19-15:21:33.515372 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-01 2024/11/19-15:21:33.515470 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00155/att-01/compact.done) = No such file or directory: 9.030 sec on http probe failed: 3, retry 2024/11/19-15:21:33.775530 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-01 2024/11/19-15:21:33.775638 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00153/att-01/compact.done) = No such file or directory: 13.044 sec on http probe failed: 5, giveup 2024/11/19-15:21:33.775666 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00153/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 229.044 sec, real 13.359 (submit 307.674 ms wait 13.044 s), input 229.044 MiB, send shutdown 2024/11/19-15:21:33.792187 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-01 2024/11/19-15:21:33.792469 139851169007360 [dcompact_etcd.cc:1773] job-00153/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:33.884257 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-01 2024/11/19-15:21:33.884368 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00154/att-01/compact.done) = No such file or directory: 13.045 sec on http probe failed: 5, giveup 2024/11/19-15:21:33.884395 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00154/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 233.065 sec, real 13.346 (submit 290.311 ms wait 13.045 s), input 233.065 MiB, send shutdown 2024/11/19-15:21:33.892198 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-01 2024/11/19-15:21:33.892461 139851179493120 [dcompact_etcd.cc:1773] job-00154/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:33.972849 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-00 2024/11/19-15:21:33.972934 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00162/att-00/compact.done) = No such file or directory: 9.216 sec on http probe failed: 4, retry 2024/11/19-15:21:34.106156 139851169007360 [dcompact_etcd.cc:1163] job-00153/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.313 sec, input = 229.044 MiB 2024/11/19-15:21:34.216947 139851179493120 [dcompact_etcd.cc:1163] job-00154/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.324 sec, input = 233.065 MiB 2024/11/19-15:21:34.384250 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-01 2024/11/19-15:21:34.384363 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00156/att-01/compact.done) = No such file or directory: 7.770 sec on http probe failed: 3, retry 2024/11/19-15:21:34.703969 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-00 2024/11/19-15:21:34.704061 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00163/att-00/compact.done) = No such file or directory: 7.205 sec on http probe failed: 3, retry 2024/11/19-15:21:34.716338 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004941.sst): old prealloc_size = 147639500, real_size = 102572000 2024/11/19-15:21:34.716464 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004941.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 198262 prefix = 4 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 raw-val = 0.1279 GB zip-val = 0.1022 GB avg-val = 644.92 avg-zval = 515.42 2024/11/19-15:21:34.716559 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004941.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 2.157 sec, Key+Value = 0.133 GB 2024/11/19-15:21:34.914383 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004941.sst): second pass time = 2.16's, 59.273'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.20's, 524.440'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3031.033'MB/sec (index lex order gen) rebuild zvType time = 0.16's, 0.317'MB/sec write SST data time = 0.04's, 2485.785'MB/sec dict compress time = 0.00's, 1944.801'MB/sec z-dict build time = 0.00's, sample length = 1.329'MB, throughput = 14461.812'MB/sec zip my value time = 2.16's, unzip length = 0.128'GB zip my value throughput = 59.273'MB/sec zip pipeline throughput = 59.273'MB/sec entries = 198262 avg-key = 24.00 avg-zkey = 1.69 avg-val = 644.92 avg-zval = 515.42 usrkeys = 198262 avg-key = 16.00 avg-zkey = 1.69 avg-val = 644.92 avg-zval = 515.42 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0032 GB value = 0.1279 GB dict = 1.33 MB all = 0.1326 GB } __ZipSize{ index = 0.0003 GB value = 0.1022 GB dict = 1.33 MB all = 0.1039 GB } UnZip/Zip{ index = 9.4772 value = 1.2513 dict = 1.00 all = 1.2770 } Zip/UnZip{ index = 0.1055 value = 0.7992 dict = 1.00 all = 0.7831 } ---------------------------- total value len = 8.057346 GB avg = 0.284 KB (by entry num) total key len = 0.721665 GB avg = 0.025 KB total ukey len = 0.494549 GB avg = 0.017 KB total ukey num = 0.028389464 Billion total entry num = 0.028389464 Billion write speed all = 49.335193420 MB/sec (with seq num) write speed all = 48.058876778 MB/sec (without seq num) 2024/11/19-15:21:34.932238 139851137550080 [compaction_job.cc:1948] [default] [JOB 146] Generated table #4941: 198262 keys, 103853980 bytes, temperature: kUnknown 2024/11/19-15:21:34.932295 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000894932271, "cf_name": "default", "job": 146, "event": "table_file_creation", "file_number": 4941, "file_size": 103853980, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 102187536, "index_size": 334720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4758288, "raw_average_key_size": 24, "raw_value_size": 127863759, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 198262, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000892, "oldest_key_time": 0, "file_creation_time": 1732000892, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4941, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:34.958706 139851137550080 [compaction_job.cc:703] job-00146: subcompact[0], size: 0.208945 G, files: /storage/mytopling/.rocksdb [004940,004941] 2024/11/19-15:21:34.966375 139851137550080 (Original Log Time 2024/11/19-15:21:29.771882) EVENT_LOG_v1 {"time_micros": 1732000889771858, "job": 146, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4763, 4764, 4765, 4766], "score": 1.85563, "input_data_size": 259250036, "oldest_snapshot_seqno": 129913905} 2024/11/19-15:21:34.966387 139851137550080 (Original Log Time 2024/11/19-15:21:34.965885) [compaction_job.cc:2022] [default] [JOB 146] Compacted 4@2 files to L3 => 208944661 bytes 2024/11/19-15:21:34.966391 139851137550080 (Original Log Time 2024/11/19-15:21:34.966223) [compaction_job.cc:1224] [default] compacted to: files[1 116 58 25 0 0 0] max score 6.47 (151 files need compaction), MB/sec: 50.0 rd, 40.3 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(247.2, 0.0 +0.0 blob) out(199.3 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 398846, records dropped: 0 output_compression: Snappy 2024/11/19-15:21:34.966396 139851137550080 (Original Log Time 2024/11/19-15:21:34.966251) EVENT_LOG_v1 {"time_micros": 1732000894966237, "cf": "default", "job": 146, "event": "compaction_finished", "compaction_time_micros": 5186840, "compaction_time_cpu_micros": 1554070, "output_level": 3, "num_output_files": 2, "total_output_size": 208944661, "num_input_records": 398846, "num_output_records": 398846, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 116, 58, 25, 0, 0, 0]} 2024/11/19-15:21:35.004024 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004691.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.004058 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895004055, "job": 146, "event": "table_file_deletion", "file_number": 4691} 2024/11/19-15:21:35.022644 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004692.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.022671 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895022669, "job": 146, "event": "table_file_deletion", "file_number": 4692} 2024/11/19-15:21:35.041384 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004693.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.041409 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895041406, "job": 146, "event": "table_file_deletion", "file_number": 4693} 2024/11/19-15:21:35.061289 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004694.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.061311 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895061309, "job": 146, "event": "table_file_deletion", "file_number": 4694} 2024/11/19-15:21:35.081931 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004695.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.081954 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895081951, "job": 146, "event": "table_file_deletion", "file_number": 4695} 2024/11/19-15:21:35.103051 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004696.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.103072 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895103070, "job": 146, "event": "table_file_deletion", "file_number": 4696} 2024/11/19-15:21:35.113091 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004697.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.113111 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895113109, "job": 146, "event": "table_file_deletion", "file_number": 4697} 2024/11/19-15:21:35.132399 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004698.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.132420 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895132418, "job": 146, "event": "table_file_deletion", "file_number": 4698} 2024/11/19-15:21:35.152560 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004699.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.152581 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895152579, "job": 146, "event": "table_file_deletion", "file_number": 4699} 2024/11/19-15:21:35.171473 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004700.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.171497 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895171495, "job": 146, "event": "table_file_deletion", "file_number": 4700} 2024/11/19-15:21:35.193942 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004701.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.193968 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895193965, "job": 146, "event": "table_file_deletion", "file_number": 4701} 2024/11/19-15:21:35.213130 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004702.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.213158 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895213155, "job": 146, "event": "table_file_deletion", "file_number": 4702} 2024/11/19-15:21:35.254394 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004703.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.254438 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895254433, "job": 146, "event": "table_file_deletion", "file_number": 4703} 2024/11/19-15:21:35.272135 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004704.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.272177 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895272172, "job": 146, "event": "table_file_deletion", "file_number": 4704} 2024/11/19-15:21:35.292500 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004705.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.292551 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895292547, "job": 146, "event": "table_file_deletion", "file_number": 4705} 2024/11/19-15:21:35.312685 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004706.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.312718 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895312714, "job": 146, "event": "table_file_deletion", "file_number": 4706} 2024/11/19-15:21:35.323338 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004755.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.323357 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895323355, "job": 146, "event": "table_file_deletion", "file_number": 4755} 2024/11/19-15:21:35.352767 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004756.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:35.352796 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000895352793, "job": 146, "event": "table_file_deletion", "file_number": 4756} 2024/11/19-15:21:35.353009 139851137550080 [compaction_job.cc:2437] [default] [JOB 164] Compacting 4@1 files to L2, score 6.47, subcompactions 1 : 1 2024/11/19-15:21:35.353018 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 162 Base level 1, inputs: [4838(31MB) 4840(31MB) 4843(1659KB) 4844(16MB)] 2024/11/19-15:21:35.524141 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-01 2024/11/19-15:21:35.524213 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00155/att-01/compact.done) = No such file or directory: 11.045 sec on http probe failed: 4, retry 2024/11/19-15:21:35.667906 139851137550080 [dcompact_etcd.cc:1163] job-00164/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.314 sec, input = 80.332 MiB 2024/11/19-15:21:35.978100 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-00 2024/11/19-15:21:35.978191 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00162/att-00/compact.done) = No such file or directory: 11.223 sec on http probe failed: 5, giveup 2024/11/19-15:21:35.978210 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00162/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.238 sec, real 11.496 (submit 267.638 ms wait 11.223 s), input 127.238 MiB, send shutdown 2024/11/19-15:21:35.982589 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-00 2024/11/19-15:21:36.052654 139851148035840 [dcompact_etcd.cc:1773] job-00162/att-00: CleanFiles 0.070 sec : 0.000 2024/11/19-15:21:36.349838 139851148035840 [dcompact_etcd.cc:1163] job-00162/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.297 sec, input = 127.238 MiB 2024/11/19-15:21:36.390567 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-01 2024/11/19-15:21:36.390631 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00156/att-01/compact.done) = No such file or directory: 9.782 sec on http probe failed: 4, retry 2024/11/19-15:21:36.710018 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-00 2024/11/19-15:21:36.710117 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00163/att-00/compact.done) = No such file or directory: 9.210 sec on http probe failed: 4, retry 2024/11/19-15:21:37.532759 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-01 2024/11/19-15:21:37.532837 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00155/att-01/compact.done) = No such file or directory: 13.054 sec on http probe failed: 5, giveup 2024/11/19-15:21:37.532856 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00155/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.634 sec, real 13.342 (submit 279.529 ms wait 13.054 s), input 255.634 MiB, send shutdown 2024/11/19-15:21:37.541041 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-01 2024/11/19-15:21:37.541260 139851158521600 [dcompact_etcd.cc:1773] job-00155/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:37.686518 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-00 2024/11/19-15:21:37.686625 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00164/att-00/compact.done) = No such file or directory: 2.008 sec on http probe failed: 1, retry 2024/11/19-15:21:37.810945 139851158521600 [dcompact_etcd.cc:1163] job-00155/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.270 sec, input = 255.634 MiB 2024/11/19-15:21:38.400206 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-01 2024/11/19-15:21:38.400309 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00156/att-01/compact.done) = No such file or directory: 11.788 sec on http probe failed: 5, giveup 2024/11/19-15:21:38.400329 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00156/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 149.879 sec, real 12.080 (submit 282.082 ms wait 11.788 s), input 149.879 MiB, send shutdown 2024/11/19-15:21:38.408752 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-01 2024/11/19-15:21:38.408981 139851127064320 [dcompact_etcd.cc:1773] job-00156/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:38.694082 139851127064320 [dcompact_etcd.cc:1163] job-00156/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.285 sec, input = 149.879 MiB 2024/11/19-15:21:38.715133 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-00 2024/11/19-15:21:38.715192 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00163/att-00/compact.done) = No such file or directory: 11.216 sec on http probe failed: 5, giveup 2024/11/19-15:21:38.715208 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00163/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.240 sec, real 11.536 (submit 314.305 ms wait 11.216 s), input 127.240 MiB, send shutdown 2024/11/19-15:21:38.723611 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-00 2024/11/19-15:21:38.723828 139850978166528 [dcompact_etcd.cc:1773] job-00163/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:39.018428 139850978166528 [dcompact_etcd.cc:1163] job-00163/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.294 sec, input = 127.240 MiB 2024/11/19-15:21:39.111871 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-02 2024/11/19-15:21:39.111959 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00153/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:21:39.225870 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-02 2024/11/19-15:21:39.225982 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00154/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:21:39.302089 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-00 2024/11/19-15:21:39.302172 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00164/att-00/compact.done) = No such file or directory: 3.625 sec on http probe failed: 2, retry 2024/11/19-15:21:39.539028 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-01 2024/11/19-15:21:39.539105 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00162/att-01/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:21:40.448776 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4942. Immutable memtables: 0. 2024/11/19-15:21:40.449503 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 165] Syncing log #4903 2024/11/19-15:21:40.449601 139850957195008 (Original Log Time 2024/11/19-15:21:40.449289) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:21:40.449609 139850957195008 [flush_job.cc:879] [default] [JOB 165] Flushing memtable with next log file: 4942 2024/11/19-15:21:40.449630 139850957195008 EVENT_LOG_v1 {"time_micros": 1732000900449624, "job": 165, "event": "flush_started", "num_memtables": 1, "num_entries": 1216323, "num_deletes": 0, "total_data_size": 503260311, "memory_usage": 540802440, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:21:40.449634 139850957195008 [flush_job.cc:910] [default] [JOB 165] Level-0 flush table #4943: started 2024/11/19-15:21:40.449934 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000019.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004943.sst in ConvertToSST, mem_size = 515.749 M, time = 291.195 us 2024/11/19-15:21:40.918857 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-00 2024/11/19-15:21:40.918970 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00164/att-00/compact.done) = No such file or directory: 5.241 sec on http probe failed: 3, retry 2024/11/19-15:21:40.931212 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004943.sst): fsize = 515.751 M, time(ms): open: 0.035, seek: 0.017, finish: 0.036, meta: 0.012, Flush: 0.013, sync: 481.141, close: 0.010, all: 481.264 2024/11/19-15:21:40.932339 139850957195008 [flush_job.cc:1083] [default] [JOB 165] Flush lasted 482739 microseconds, and 384628 cpu microseconds. 2024/11/19-15:21:40.951928 139850957195008 (Original Log Time 2024/11/19-15:21:40.931254) [flush_job.cc:1029] [default] [JOB 165] Level-0 flush table #4943: 540803866 bytes OK (needs compaction) 2024/11/19-15:21:40.951938 139850957195008 (Original Log Time 2024/11/19-15:21:40.932354) [memtable_list.cc:560] [default] Level-0 commit table #4943 started 2024/11/19-15:21:40.951941 139850957195008 (Original Log Time 2024/11/19-15:21:40.951851) [memtable_list.cc:759] [default] Level-0 commit table #4943: memtable #1 done 2024/11/19-15:21:40.951944 139850957195008 (Original Log Time 2024/11/19-15:21:40.951874) EVENT_LOG_v1 {"time_micros": 1732000900951869, "job": 165, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [2, 116, 58, 25, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:21:40.951948 139850957195008 (Original Log Time 2024/11/19-15:21:40.951893) [db_impl_compaction_flush.cc:310] [default] Level summary: files[2 116 58 25 0 0 0] max score 6.31 (148 files need compaction) 2024/11/19-15:21:41.121513 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-02 2024/11/19-15:21:41.121606 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00153/att-02/compact.done) = No such file or directory: 7.006 sec on http probe failed: 2, retry 2024/11/19-15:21:41.234523 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-02 2024/11/19-15:21:41.234606 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00154/att-02/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:21:41.544607 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-01 2024/11/19-15:21:41.544678 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00162/att-01/compact.done) = No such file or directory: 5.189 sec on http probe failed: 2, retry 2024/11/19-15:21:42.206292 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-01 2024/11/19-15:21:42.206384 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00163/att-01/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:21:42.450362 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-02 2024/11/19-15:21:42.450482 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00156/att-02/compact.done) = No such file or directory: 3.747 sec on http probe failed: 1, retry 2024/11/19-15:21:42.536034 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-00 2024/11/19-15:21:42.536108 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00164/att-00/compact.done) = No such file or directory: 6.858 sec on http probe failed: 4, retry 2024/11/19-15:21:42.818214 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-02 2024/11/19-15:21:42.818325 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00155/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:21:43.127530 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-02 2024/11/19-15:21:43.127626 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00153/att-02/compact.done) = No such file or directory: 9.016 sec on http probe failed: 3, retry 2024/11/19-15:21:43.240486 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-02 2024/11/19-15:21:43.240576 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00154/att-02/compact.done) = No such file or directory: 9.018 sec on http probe failed: 3, retry 2024/11/19-15:21:43.550073 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-01 2024/11/19-15:21:43.550166 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00162/att-01/compact.done) = No such file or directory: 7.195 sec on http probe failed: 3, retry 2024/11/19-15:21:44.147770 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-00 2024/11/19-15:21:44.147875 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00164/att-00/compact.done) = No such file or directory: 8.475 sec on http probe failed: 5, giveup 2024/11/19-15:21:44.147893 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00164/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 80.332 sec, real 8.794 (submit 314.437 ms wait 8.475 s), input 80.332 MiB, send shutdown 2024/11/19-15:21:44.154599 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-00 2024/11/19-15:21:44.154824 139851137550080 [dcompact_etcd.cc:1773] job-00164/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:44.214626 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-01 2024/11/19-15:21:44.214686 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00163/att-01/compact.done) = No such file or directory: 5.188 sec on http probe failed: 2, retry 2024/11/19-15:21:44.394104 139851137550080 [dcompact_etcd.cc:1163] job-00164/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.239 sec, input = 80.332 MiB 2024/11/19-15:21:44.459455 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-02 2024/11/19-15:21:44.459542 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00156/att-02/compact.done) = No such file or directory: 5.756 sec on http probe failed: 2, retry 2024/11/19-15:21:44.828134 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-02 2024/11/19-15:21:44.828219 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00155/att-02/compact.done) = No such file or directory: 7.007 sec on http probe failed: 2, retry 2024/11/19-15:21:45.133612 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-02 2024/11/19-15:21:45.133699 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00153/att-02/compact.done) = No such file or directory: 11.022 sec on http probe failed: 4, retry 2024/11/19-15:21:45.245257 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-02 2024/11/19-15:21:45.245340 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00154/att-02/compact.done) = No such file or directory: 11.024 sec on http probe failed: 4, retry 2024/11/19-15:21:45.558897 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-01 2024/11/19-15:21:45.558992 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00162/att-01/compact.done) = No such file or directory: 9.200 sec on http probe failed: 4, retry 2024/11/19-15:21:46.223243 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-01 2024/11/19-15:21:46.223331 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00163/att-01/compact.done) = No such file or directory: 7.196 sec on http probe failed: 3, retry 2024/11/19-15:21:46.407836 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-01 2024/11/19-15:21:46.407934 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00164/att-01/compact.done) = No such file or directory: 2.008 sec on http probe failed: 1, retry 2024/11/19-15:21:46.468571 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-02 2024/11/19-15:21:46.468637 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00156/att-02/compact.done) = No such file or directory: 7.766 sec on http probe failed: 3, retry 2024/11/19-15:21:46.833172 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-02 2024/11/19-15:21:46.833268 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00155/att-02/compact.done) = No such file or directory: 9.017 sec on http probe failed: 3, retry 2024/11/19-15:21:47.142052 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-02 2024/11/19-15:21:47.142131 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00153/att-02/compact.done) = No such file or directory: 13.028 sec on http probe failed: 5, giveup 2024/11/19-15:21:47.142149 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00153/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 229.044 sec, real 13.349 (submit 313.496 ms wait 13.028 s), input 229.044 MiB, send shutdown 2024/11/19-15:21:47.147010 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00153/att-02 2024/11/19-15:21:47.147222 139851169007360 [dcompact_etcd.cc:1773] job-00153/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:47.149190 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00153) = success 2024/11/19-15:21:47.149198 139851169007360 [dcompact_etcd.cc:1773] job-00153/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:21:47.149217 139851169007360 (Original Log Time 2024/11/19-15:21:07.034984) EVENT_LOG_v1 {"time_micros": 1732000867034978, "job": 153, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4600, 4601, 4770, 4771], "score": 1.90674, "input_data_size": 198655776, "oldest_snapshot_seqno": 130822884} 2024/11/19-15:21:47.149224 139851169007360 [compaction_job.cc:2437] [default] [JOB 153] Compacting 4@2 files to L3, score 1.91, subcompactions 1 : 1 2024/11/19-15:21:47.149233 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 151 Base level 2, inputs: [4600(52MB) 4601(30MB) 4770(52MB) 4771(52MB)] 2024/11/19-15:21:47.149483 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004944.sst) /storage/mytopling/..//tmp/Topling-82038-mr9Ogv 2024/11/19-15:21:47.250336 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-02 2024/11/19-15:21:47.250429 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00154/att-02/compact.done) = No such file or directory: 13.028 sec on http probe failed: 5, giveup 2024/11/19-15:21:47.250446 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00154/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 233.065 sec, real 13.358 (submit 324.336 ms wait 13.028 s), input 233.065 MiB, send shutdown 2024/11/19-15:21:47.255865 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00154/att-02 2024/11/19-15:21:47.256090 139851179493120 [dcompact_etcd.cc:1773] job-00154/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:47.258056 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00154) = success 2024/11/19-15:21:47.258064 139851179493120 [dcompact_etcd.cc:1773] job-00154/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:21:47.258084 139851179493120 (Original Log Time 2024/11/19-15:21:07.199443) EVENT_LOG_v1 {"time_micros": 1732000867199437, "job": 154, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4773, 4775, 4772, 4774], "score": 1.75762, "input_data_size": 202143904, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:47.258091 139851179493120 [compaction_job.cc:2437] [default] [JOB 154] Compacting 4@2 files to L3, score 1.76, subcompactions 1 : 1 2024/11/19-15:21:47.258100 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 152 Base level 2, inputs: [4773(52MB) 4775(52MB) 4772(52MB) 4774(34MB)] 2024/11/19-15:21:47.258368 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004945.sst) /storage/mytopling/..//tmp/Topling-82038-s6GzC1 2024/11/19-15:21:47.490692 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004944.sst): first pass time = 0.34's, 393.375'MB/sec 2024/11/19-15:21:47.490694 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004944.sst: sumWaitingMem = 0.012 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0120 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:21:47.490728 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004944.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.012 GB, dictZip workingMem = 0.0065 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:21:47.563159 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004944.sst): index pass time = 0.07's, 204.772'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 926921 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:21:47.568185 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-01 2024/11/19-15:21:47.568259 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00162/att-01/compact.done) = No such file or directory: 11.209 sec on http probe failed: 5, giveup 2024/11/19-15:21:47.568281 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00162/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.238 sec, real 11.515 (submit 297.032 ms wait 11.209 s), input 127.238 MiB, send shutdown 2024/11/19-15:21:47.576168 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-01 2024/11/19-15:21:47.576393 139851148035840 [dcompact_etcd.cc:1773] job-00162/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:47.587433 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004945.sst): first pass time = 0.33's, 407.891'MB/sec 2024/11/19-15:21:47.587438 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004945.sst: sumWaitingMem = 0.012 GB, sumWorkingMem = 0.006 GB, nltTrie workingMem = 0.0120 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:21:47.587471 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004945.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.019 GB, dictZip workingMem = 0.0065 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:21:47.681166 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004945.sst): index pass time = 0.09's, 158.284'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 926870 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:21:47.888233 139851148035840 [dcompact_etcd.cc:1163] job-00162/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.312 sec, input = 127.238 MiB 2024/11/19-15:21:48.023745 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-01 2024/11/19-15:21:48.023865 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00164/att-01/compact.done) = No such file or directory: 3.621 sec on http probe failed: 2, retry 2024/11/19-15:21:48.233675 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-01 2024/11/19-15:21:48.233805 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00163/att-01/compact.done) = No such file or directory: 9.205 sec on http probe failed: 4, retry 2024/11/19-15:21:48.484523 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-02 2024/11/19-15:21:48.484625 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00156/att-02/compact.done) = No such file or directory: 9.776 sec on http probe failed: 4, retry 2024/11/19-15:21:48.842837 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-02 2024/11/19-15:21:48.842956 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00155/att-02/compact.done) = No such file or directory: 11.022 sec on http probe failed: 4, retry 2024/11/19-15:21:49.587519 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004945.sst): old prealloc_size = 147639500, real_size = 36285424 2024/11/19-15:21:49.587964 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004945.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926870 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0343 GB avg-val = 116.81 avg-zval = 37.01 2024/11/19-15:21:49.588029 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004945.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 2.001 sec, Key+Value = 0.134 GB 2024/11/19-15:21:49.609594 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004944.sst): old prealloc_size = 147639500, real_size = 36422944 2024/11/19-15:21:49.610030 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004944.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926921 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.80 avg-zval = 37.15 2024/11/19-15:21:49.610083 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004944.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 2.119 sec, Key+Value = 0.134 GB 2024/11/19-15:21:49.639185 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-01 2024/11/19-15:21:49.639272 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00164/att-01/compact.done) = No such file or directory: 5.236 sec on http probe failed: 3, retry 2024/11/19-15:21:50.116788 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004945.sst): second pass time = 2.00's, 54.132'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.53's, 70.181'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 4094.893'MB/sec (index lex order gen) rebuild zvType time = 0.43's, 0.533'MB/sec write SST data time = 0.09's, 395.277'MB/sec dict compress time = 0.00's, 1479.837'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18992.299'MB/sec zip my value time = 2.00's, unzip length = 0.108'GB zip my value throughput = 54.132'MB/sec zip pipeline throughput = 100.759'MB/sec entries = 926870 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 37.01 usrkeys = 926870 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 37.01 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0343 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5807 value = 3.1563 dict = 1.00 all = 3.6135 } Zip/UnZip{ index = 0.0945 value = 0.3168 dict = 1.00 all = 0.2767 } ---------------------------- total value len = 8.165611 GB avg = 0.279 KB (by entry num) total key len = 0.747617 GB avg = 0.026 KB total ukey len = 0.513087 GB avg = 0.018 KB total ukey num = 0.029316334 Billion total entry num = 0.029316334 Billion write speed all = 46.146997887 MB/sec (with seq num) write speed all = 44.932748053 MB/sec (without seq num) 2024/11/19-15:21:50.117777 139851179493120 [compaction_job.cc:1948] [default] [JOB 154] Generated table #4945: 926870 keys, 37143477 bytes, temperature: kUnknown 2024/11/19-15:21:50.117829 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000910117804, "cf_name": "default", "job": 154, "event": "table_file_creation", "file_number": 4945, "file_size": 37143477, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34301536, "index_size": 1752000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25952360, "raw_average_key_size": 28, "raw_value_size": 108265508, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926870, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000907, "oldest_key_time": 0, "file_creation_time": 1732000907, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4945, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:50.137576 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004946.sst) /storage/mytopling/..//tmp/Topling-82038-Wvrnb0 2024/11/19-15:21:50.138324 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004944.sst): second pass time = 2.12's, 51.096'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.53's, 70.513'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 4121.814'MB/sec (index lex order gen) rebuild zvType time = 0.43's, 0.534'MB/sec write SST data time = 0.09's, 393.634'MB/sec dict compress time = 0.00's, 1494.540'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18465.367'MB/sec zip my value time = 2.12's, unzip length = 0.108'GB zip my value throughput = 51.096'MB/sec zip pipeline throughput = 102.193'MB/sec entries = 926921 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.80 avg-zval = 37.15 usrkeys = 926921 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.80 avg-zval = 37.15 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5824 value = 3.1436 dict = 1.00 all = 3.6001 } Zip/UnZip{ index = 0.0945 value = 0.3181 dict = 1.00 all = 0.2778 } ---------------------------- total value len = 8.273875 GB avg = 0.274 KB (by entry num) total key len = 0.773571 GB avg = 0.026 KB total ukey len = 0.531625 GB avg = 0.018 KB total ukey num = 0.030243255 Billion total entry num = 0.030243255 Billion write speed all = 46.836668688 MB/sec (with seq num) write speed all = 45.584166495 MB/sec (without seq num) 2024/11/19-15:21:50.139259 139851169007360 [compaction_job.cc:1948] [default] [JOB 153] Generated table #4944: 926921 keys, 37281169 bytes, temperature: kUnknown 2024/11/19-15:21:50.139305 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000910139283, "cf_name": "default", "job": 153, "event": "table_file_creation", "file_number": 4944, "file_size": 37281169, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34439248, "index_size": 1751808, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953788, "raw_average_key_size": 28, "raw_value_size": 108263998, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926921, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000907, "oldest_key_time": 0, "file_creation_time": 1732000907, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4944, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:50.159422 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004947.sst) /storage/mytopling/..//tmp/Topling-82038-QYEYcG 2024/11/19-15:21:50.242810 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-01 2024/11/19-15:21:50.242940 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00163/att-01/compact.done) = No such file or directory: 11.215 sec on http probe failed: 5, giveup 2024/11/19-15:21:50.242965 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00163/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.240 sec, real 11.519 (submit 294.497 ms wait 11.215 s), input 127.240 MiB, send shutdown 2024/11/19-15:21:50.250030 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-01 2024/11/19-15:21:50.250298 139850978166528 [dcompact_etcd.cc:1773] job-00163/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:50.413678 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004946.sst): first pass time = 0.28's, 399.038'MB/sec 2024/11/19-15:21:50.413683 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004946.sst: sumWaitingMem = 0.010 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0099 GB, waited 0.000 sec, Key+Value = 0.110 GB 2024/11/19-15:21:50.413721 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004946.sst: sumWaitingMem = 0.005 GB, sumWorkingMem = 0.010 GB, dictZip workingMem = 0.0053 GB, waited 0.000 sec, Key+Value = 0.110 GB 2024/11/19-15:21:50.442280 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004947.sst): first pass time = 0.28's, 374.601'MB/sec 2024/11/19-15:21:50.442283 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004947.sst: sumWaitingMem = 0.010 GB, sumWorkingMem = 0.015 GB, nltTrie workingMem = 0.0095 GB, waited 0.000 sec, Key+Value = 0.106 GB 2024/11/19-15:21:50.442318 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004947.sst: sumWaitingMem = 0.005 GB, sumWorkingMem = 0.025 GB, dictZip workingMem = 0.0051 GB, waited 0.000 sec, Key+Value = 0.106 GB 2024/11/19-15:21:50.480522 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004946.sst): index pass time = 0.07's, 182.268'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 760964 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0122 GB zip-key = 0.0014 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:21:50.494083 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-02 2024/11/19-15:21:50.494153 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00156/att-02/compact.done) = No such file or directory: 11.791 sec on http probe failed: 5, giveup 2024/11/19-15:21:50.494172 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00156/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 149.879 sec, real 12.085 (submit 285.018 ms wait 11.791 s), input 149.879 MiB, send shutdown 2024/11/19-15:21:50.497577 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004947.sst): index pass time = 0.06's, 211.870'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 731696 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0117 GB zip-key = 0.0014 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:21:50.501513 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00156/att-02 2024/11/19-15:21:50.501732 139851127064320 [dcompact_etcd.cc:1773] job-00156/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:50.503583 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00156) = success 2024/11/19-15:21:50.503596 139851127064320 [dcompact_etcd.cc:1773] job-00156/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:21:50.503618 139851127064320 (Original Log Time 2024/11/19-15:21:14.226678) EVENT_LOG_v1 {"time_micros": 1732000874226670, "job": 156, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4769, 4602, 4603, 4606], "score": 1.52202, "input_data_size": 145850606, "oldest_snapshot_seqno": 131139990} 2024/11/19-15:21:50.503626 139851127064320 [compaction_job.cc:2437] [default] [JOB 156] Compacting 4@2 files to L3, score 1.52, subcompactions 1 : 1 2024/11/19-15:21:50.503635 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 154 Base level 2, inputs: [4769(38MB) 4602(6826KB) 4603(31MB) 4606(62MB)] 2024/11/19-15:21:50.503869 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004948.sst) /storage/mytopling/..//tmp/Topling-82038-DglCoY 2024/11/19-15:21:50.529906 139850978166528 [dcompact_etcd.cc:1163] job-00163/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.279 sec, input = 127.240 MiB 2024/11/19-15:21:50.626330 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004948.sst): first pass time = 0.12's, 396.104'MB/sec 2024/11/19-15:21:50.626335 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004948.sst: sumWaitingMem = 0.004 GB, sumWorkingMem = 0.010 GB, nltTrie workingMem = 0.0044 GB, waited 0.000 sec, Key+Value = 0.049 GB 2024/11/19-15:21:50.626371 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004948.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.015 GB, dictZip workingMem = 0.0024 GB, waited 0.000 sec, Key+Value = 0.049 GB 2024/11/19-15:21:50.667042 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004948.sst): index pass time = 0.04's, 131.834'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 335080 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0054 GB zip-key = 0.0006 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:21:50.861072 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-02 2024/11/19-15:21:50.861201 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00155/att-02/compact.done) = No such file or directory: 13.032 sec on http probe failed: 5, giveup 2024/11/19-15:21:50.861239 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00155/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.634 sec, real 13.320 (submit 269.586 ms wait 13.032 s), input 255.634 MiB, send shutdown 2024/11/19-15:21:50.878397 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00155/att-02 2024/11/19-15:21:50.878691 139851158521600 [dcompact_etcd.cc:1773] job-00155/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:50.880663 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00155) = success 2024/11/19-15:21:50.880682 139851158521600 [dcompact_etcd.cc:1773] job-00155/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:21:50.880711 139851158521600 (Original Log Time 2024/11/19-15:21:10.685462) EVENT_LOG_v1 {"time_micros": 1732000870685453, "job": 155, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4779, 4780, 4781, 4782], "score": 1.60589, "input_data_size": 221689376, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:50.880722 139851158521600 [compaction_job.cc:2437] [default] [JOB 155] Compacting 4@2 files to L3, score 1.61, subcompactions 1 : 1 2024/11/19-15:21:50.880734 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 153 Base level 2, inputs: [4779(52MB) 4780(52MB) 4781(52MB) 4782(52MB)] 2024/11/19-15:21:50.881059 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004949.sst) /storage/mytopling/..//tmp/Topling-82038-hJgJrv 2024/11/19-15:21:51.088027 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-02 2024/11/19-15:21:51.088135 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00162/att-02/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:21:51.258292 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-01 2024/11/19-15:21:51.258409 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00164/att-01/compact.done) = No such file or directory: 6.852 sec on http probe failed: 4, retry 2024/11/19-15:21:51.424465 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004948.sst): old prealloc_size = 147639500, real_size = 13325552 2024/11/19-15:21:51.424677 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004948.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 335080 prefix = 4 raw-key = 0.0054 GB zip-key = 0.0006 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0391 GB zip-val = 0.0126 GB avg-val = 116.75 avg-zval = 37.62 2024/11/19-15:21:51.424759 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004948.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.010 GB, reorder workingMem = 0.0008 GB, waited 0.798 sec, Key+Value = 0.049 GB 2024/11/19-15:21:51.564255 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004949.sst: sumWaitingMem = 0.012 GB, sumWorkingMem = 0.010 GB, nltTrie workingMem = 0.0120 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:21:51.571955 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004949.sst): first pass time = 0.69's, 194.272'MB/sec 2024/11/19-15:21:51.571986 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004949.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.023 GB, dictZip workingMem = 0.0065 GB, waited 0.008 sec, Key+Value = 0.134 GB 2024/11/19-15:21:51.737133 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004948.sst): second pass time = 0.80's, 49.019'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.31's, 43.652'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3230.714'MB/sec (index lex order gen) rebuild zvType time = 0.25's, 0.340'MB/sec write SST data time = 0.07's, 206.578'MB/sec dict compress time = 0.00's, 1154.200'MB/sec z-dict build time = 0.00's, sample length = 0.402'MB, throughput = 5069.109'MB/sec zip my value time = 0.80's, unzip length = 0.039'GB zip my value throughput = 49.019'MB/sec zip pipeline throughput = 63.134'MB/sec entries = 335080 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.75 avg-zval = 37.62 usrkeys = 335080 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.75 avg-zval = 37.62 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0067 GB value = 0.0391 GB dict = 0.40 MB all = 0.0485 GB } __ZipSize{ index = 0.0006 GB value = 0.0126 GB dict = 0.40 MB all = 0.0136 GB } UnZip/Zip{ index = 10.5621 value = 3.1030 dict = 1.00 all = 3.5539 } Zip/UnZip{ index = 0.0947 value = 0.3223 dict = 1.00 all = 0.2814 } ---------------------------- total value len = 8.312995 GB avg = 0.272 KB (by entry num) total key len = 0.782953 GB avg = 0.026 KB total ukey len = 0.538327 GB avg = 0.018 KB total ukey num = 0.030578335 Billion total entry num = 0.030578335 Billion write speed all = 46.701226987 MB/sec (with seq num) write speed all = 45.445242989 MB/sec (without seq num) 2024/11/19-15:21:51.739970 139851127064320 [compaction_job.cc:1948] [default] [JOB 156] Generated table #4948: 335080 keys, 13647467 bytes, temperature: kUnknown 2024/11/19-15:21:51.740037 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000911740006, "cf_name": "default", "job": 156, "event": "table_file_creation", "file_number": 4948, "file_size": 13647467, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 12607072, "index_size": 634496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 9382240, "raw_average_key_size": 28, "raw_value_size": 39120088, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 335080, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000910, "oldest_key_time": 0, "file_creation_time": 1732000910, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4948, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:51.762652 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004950.sst) /storage/mytopling/..//tmp/Topling-82038-3tq9GT 2024/11/19-15:21:51.811003 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004949.sst): index pass time = 0.25's, 60.114'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 926909 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:21:51.843780 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004950.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.017 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.008 GB 2024/11/19-15:21:51.843885 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004950.sst): index pass time = 0.00's,5594.249'MB/sec index type = UintIndex_AllOne usrkeys = 100000 min-keylen = 4 max-keylen = 4 prefix = 4 subcplen = 1 raw-key = 0.0004 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 0.00 2024/11/19-15:21:51.843940 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004950.sst): first pass time = 0.08's, 102.149'MB/sec 2024/11/19-15:21:51.843957 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004950.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.017 GB, dictZip workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.008 GB 2024/11/19-15:21:52.290855 139851127064320 [top_zip_table_builder.cc:2752] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004950.sst): part-0: reject ZipStore ratio = 1.032 2024/11/19-15:21:52.352613 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004950.sst): old prealloc_size = 147639500, real_size = 6797040 2024/11/19-15:21:52.352664 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004950.sst): index type = UintIndex_AllOne , store type = ZipOffsetBlobStore usrkeys = 100000 prefix = 4 raw-key = 0.0004 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 0.00 raw-val = 0.0067 GB zip-val = 0.0068 GB avg-val = 67.01 avg-zval = 67.72 2024/11/19-15:21:52.365869 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004950.sst): second pass time = 0.45's, 14.995'MB/sec, value only(80.7% of KV) wait indexing time = 0.06's, remap KeyValue time = 0.01's, 510.859'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 6.203'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 5.041'MB/sec write SST data time = 0.01's, 819.090'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = 0.000'MB/sec zip my value time = 0.45's, unzip length = 0.007'GB zip my value throughput = 14.995'MB/sec zip pipeline throughput = 84.562'MB/sec entries = 100000 avg-key = 16.00 avg-zkey = 0.00 avg-val = 67.01 avg-zval = 67.72 usrkeys = 100000 avg-key = 8.00 avg-zkey = 0.00 avg-val = 67.01 avg-zval = 67.72 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0008 GB value = 0.0067 GB dict = 0.00 MB all = 0.0083 GB } __ZipSize{ index = 0.0000 GB value = 0.0068 GB dict = 0.00 MB all = 0.0068 GB } UnZip/Zip{ index =4166.6667 value = 0.9896 dict = 0.00 all = 1.2256 } Zip/UnZip{ index = 0.0002 value = 1.0105 dict = 0.00 all = 0.8159 } ---------------------------- total value len = 8.319697 GB avg = 0.271 KB (by entry num) total key len = 0.784553 GB avg = 0.026 KB total ukey len = 0.539127 GB avg = 0.018 KB total ukey num = 0.030678335 Billion total entry num = 0.030678335 Billion write speed all = 46.593434713 MB/sec (with seq num) write speed all = 45.337398032 MB/sec (without seq num) 2024/11/19-15:21:52.366502 139851127064320 [compaction_job.cc:1948] [default] [JOB 156] Generated table #4950: 100000 keys, 6772998 bytes, temperature: kUnknown 2024/11/19-15:21:52.366549 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000912366527, "cf_name": "default", "job": 156, "event": "table_file_creation", "file_number": 4950, "file_size": 6772998, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 6771712, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1600000, "raw_average_key_size": 16, "raw_value_size": 6701254, "raw_average_value_size": 67, "num_data_blocks": 1, "num_entries": 100000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-ZipOffsetBlobStore", "creation_time": 1732000911, "oldest_key_time": 0, "file_creation_time": 1732000911, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4950, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:52.369212 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004951.sst) /storage/mytopling/..//tmp/Topling-82038-MbQ2qD 2024/11/19-15:21:52.453470 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004951.sst): first pass time = 0.08's,1191.287'MB/sec 2024/11/19-15:21:52.453473 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004951.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.017 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.100 GB 2024/11/19-15:21:52.453519 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004951.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.017 GB, dictZip workingMem = 0.0061 GB, waited 0.000 sec, Key+Value = 0.100 GB 2024/11/19-15:21:52.453597 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004951.sst): index pass time = 0.00's,9277.220'MB/sec index type = UintIndex_AllOne usrkeys = 96773 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0008 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:21:52.465849 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004946.sst): old prealloc_size = 147639500, real_size = 29933360 2024/11/19-15:21:52.466234 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004946.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 760964 prefix = 4 raw-key = 0.0122 GB zip-key = 0.0014 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0889 GB zip-val = 0.0283 GB avg-val = 116.77 avg-zval = 37.18 2024/11/19-15:21:52.466294 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004946.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0019 GB, waited 2.053 sec, Key+Value = 0.110 GB 2024/11/19-15:21:52.470186 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004947.sst): old prealloc_size = 147639500, real_size = 28733056 2024/11/19-15:21:52.470701 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004947.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 731696 prefix = 4 raw-key = 0.0117 GB zip-key = 0.0014 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0855 GB zip-val = 0.0272 GB avg-val = 116.80 avg-zval = 37.12 2024/11/19-15:21:52.470798 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004947.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.014 GB, reorder workingMem = 0.0018 GB, waited 2.029 sec, Key+Value = 0.106 GB 2024/11/19-15:21:52.878046 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-01 2024/11/19-15:21:52.878142 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00164/att-01/compact.done) = No such file or directory: 8.471 sec on http probe failed: 5, giveup 2024/11/19-15:21:52.878174 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00164/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 80.332 sec, real 8.723 (submit 239.195 ms wait 8.471 s), input 80.332 MiB, send shutdown 2024/11/19-15:21:52.887850 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-01 2024/11/19-15:21:52.904991 139851137550080 [dcompact_etcd.cc:1773] job-00164/att-01: CleanFiles 0.017 sec : 0.000 2024/11/19-15:21:53.100399 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-02 2024/11/19-15:21:53.100505 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00162/att-02/compact.done) = No such file or directory: 5.200 sec on http probe failed: 2, retry 2024/11/19-15:21:53.230425 139851137550080 [dcompact_etcd.cc:1163] job-00164/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.325 sec, input = 80.332 MiB 2024/11/19-15:21:53.340969 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004947.sst): second pass time = 2.03's, 42.191'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 33.780'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2781.597'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.266'MB/sec write SST data time = 0.18's, 160.345'MB/sec dict compress time = 0.00's, 1325.523'MB/sec z-dict build time = 0.00's, sample length = 0.856'MB, throughput = 13729.835'MB/sec zip my value time = 2.03's, unzip length = 0.085'GB zip my value throughput = 42.191'MB/sec zip pipeline throughput = 86.482'MB/sec entries = 731696 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 37.12 usrkeys = 731696 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 37.12 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 10 UnZipSize{ index = 0.0146 GB value = 0.0855 GB dict = 0.86 MB all = 0.1060 GB } __ZipSize{ index = 0.0014 GB value = 0.0272 GB dict = 0.86 MB all = 0.0294 GB } UnZip/Zip{ index = 10.5109 value = 3.1470 dict = 1.00 all = 3.6021 } Zip/UnZip{ index = 0.0951 value = 0.3178 dict = 1.00 all = 0.2776 } ---------------------------- total value len = 8.405162 GB avg = 0.268 KB (by entry num) total key len = 0.805041 GB avg = 0.026 KB total ukey len = 0.553761 GB avg = 0.018 KB total ukey num = 0.031410031 Billion total entry num = 0.031410031 Billion write speed all = 46.901627281 MB/sec (with seq num) write speed all = 45.622018861 MB/sec (without seq num) 2024/11/19-15:21:53.361282 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004946.sst): second pass time = 2.05's, 43.322'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 34.220'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3904.895'MB/sec (index lex order gen) rebuild zvType time = 0.59's, 0.322'MB/sec write SST data time = 0.30's, 101.022'MB/sec dict compress time = 0.00's, 1430.213'MB/sec z-dict build time = 0.00's, sample length = 0.889'MB, throughput = 14699.686'MB/sec zip my value time = 2.05's, unzip length = 0.089'GB zip my value throughput = 43.322'MB/sec zip pipeline throughput = 97.682'MB/sec entries = 760964 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.77 avg-zval = 37.18 usrkeys = 760964 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.77 avg-zval = 37.18 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0152 GB value = 0.0889 GB dict = 0.89 MB all = 0.1102 GB } __ZipSize{ index = 0.0014 GB value = 0.0283 GB dict = 0.89 MB all = 0.0306 GB } UnZip/Zip{ index = 10.5101 value = 3.1405 dict = 1.00 all = 3.5955 } Zip/UnZip{ index = 0.0951 value = 0.3184 dict = 1.00 all = 0.2781 } ---------------------------- total value len = 8.494023 GB avg = 0.264 KB (by entry num) total key len = 0.826348 GB avg = 0.026 KB total ukey len = 0.568980 GB avg = 0.018 KB total ukey num = 0.032170995 Billion total entry num = 0.032170995 Billion write speed all = 47.457731718 MB/sec (with seq num) write speed all = 46.147258130 MB/sec (without seq num) 2024/11/19-15:21:53.530418 139851169007360 [compaction_job.cc:1948] [default] [JOB 153] Generated table #4947: 731696 keys, 29414173 bytes, temperature: kUnknown 2024/11/19-15:21:53.530498 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000913530464, "cf_name": "default", "job": 153, "event": "table_file_creation", "file_number": 4947, "file_size": 29414173, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 27157680, "index_size": 1392256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 20487488, "raw_average_key_size": 28, "raw_value_size": 85464884, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 731696, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000910, "oldest_key_time": 0, "file_creation_time": 1732000910, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4947, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:53.615975 139851179493120 [compaction_job.cc:1948] [default] [JOB 154] Generated table #4946: 760964 keys, 30640647 bytes, temperature: kUnknown 2024/11/19-15:21:53.616051 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000913616018, "cf_name": "default", "job": 154, "event": "table_file_creation", "file_number": 4946, "file_size": 30640647, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 28294880, "index_size": 1448064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 21306992, "raw_average_key_size": 28, "raw_value_size": 88861284, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 760964, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000910, "oldest_key_time": 0, "file_creation_time": 1732000910, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4946, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:53.727540 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-02 2024/11/19-15:21:53.727949 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00163/att-02/compact.done) = No such file or directory: 3.185 sec on http probe failed: 1, retry 2024/11/19-15:21:53.848109 139851179493120 [compaction_job.cc:703] job-00154: subcompact[0], size: 0.067784 G, files: /storage/mytopling/.rocksdb [004945,004946] 2024/11/19-15:21:53.858363 139851169007360 [compaction_job.cc:703] job-00153: subcompact[0], size: 0.066695 G, files: /storage/mytopling/.rocksdb [004944,004947] 2024/11/19-15:21:53.862518 139851179493120 (Original Log Time 2024/11/19-15:21:47.258133) EVENT_LOG_v1 {"time_micros": 1732000907258112, "job": 154, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4773, 4775, 4772, 4774], "score": 1.75762, "input_data_size": 202143904, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:53.862544 139851179493120 (Original Log Time 2024/11/19-15:21:53.861361) [compaction_job.cc:2022] [default] [JOB 154] Compacted 4@2 files to L3 => 67784124 bytes 2024/11/19-15:21:53.862555 139851179493120 (Original Log Time 2024/11/19-15:21:53.862173) [compaction_job.cc:1224] [default] compacted to: files[2 116 54 27 0 0 0] max score 6.31 (148 files need compaction), MB/sec: 30.7 rd, 10.3 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(192.8, 0.0 +0.0 blob) out(64.6 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 1687834, records dropped: 0 output_compression: Snappy 2024/11/19-15:21:53.862590 139851179493120 (Original Log Time 2024/11/19-15:21:53.862242) EVENT_LOG_v1 {"time_micros": 1732000913862208, "cf": "default", "job": 154, "event": "compaction_finished", "compaction_time_micros": 6590034, "compaction_time_cpu_micros": 3816526, "output_level": 3, "num_output_files": 2, "total_output_size": 67784124, "num_input_records": 1687834, "num_output_records": 1687834, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 116, 54, 27, 0, 0, 0]} 2024/11/19-15:21:53.862782 139851179493120 [compaction_job.cc:2437] [default] [JOB 166] Compacting 3@1 files to L2, score 6.31, subcompactions 1 : 1 2024/11/19-15:21:53.862804 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 164 Base level 1, inputs: [4817(21MB) 4826(3810KB) 4831(21MB)] 2024/11/19-15:21:53.866127 139851169007360 (Original Log Time 2024/11/19-15:21:47.149263) EVENT_LOG_v1 {"time_micros": 1732000907149245, "job": 153, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4600, 4601, 4770, 4771], "score": 1.90674, "input_data_size": 198655776, "oldest_snapshot_seqno": 130822884} 2024/11/19-15:21:53.866142 139851169007360 (Original Log Time 2024/11/19-15:21:53.865515) [compaction_job.cc:2022] [default] [JOB 153] Compacted 4@2 files to L3 => 66695342 bytes 2024/11/19-15:21:53.866156 139851169007360 (Original Log Time 2024/11/19-15:21:53.865976) [compaction_job.cc:1224] [default] compacted to: files[2 116 50 29 0 0 0] max score 6.22 (145 files need compaction), MB/sec: 29.6 rd, 9.9 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(189.5, 0.0 +0.0 blob) out(63.6 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 1658617, records dropped: 0 output_compression: Snappy 2024/11/19-15:21:53.866180 139851169007360 (Original Log Time 2024/11/19-15:21:53.866008) EVENT_LOG_v1 {"time_micros": 1732000913865992, "cf": "default", "job": 153, "event": "compaction_finished", "compaction_time_micros": 6709144, "compaction_time_cpu_micros": 3798786, "output_level": 3, "num_output_files": 2, "total_output_size": 66695342, "num_input_records": 1658617, "num_output_records": 1658617, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 116, 50, 29, 0, 0, 0]} 2024/11/19-15:21:53.876989 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004757.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:53.877049 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000913877042, "job": 153, "event": "table_file_deletion", "file_number": 4757} 2024/11/19-15:21:53.919025 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004761.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:53.919077 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000913919073, "job": 153, "event": "table_file_deletion", "file_number": 4761} 2024/11/19-15:21:53.919326 139851169007360 [compaction_job.cc:2437] [default] [JOB 167] Compacting 4@1 files to L2, score 6.22, subcompactions 1 : 1 2024/11/19-15:21:53.919343 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 165 Base level 1, inputs: [4841(31MB) 4846(31MB) 4849(31MB) 4853(31MB)] 2024/11/19-15:21:54.120252 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004951.sst): old prealloc_size = 147639500, real_size = 30055368 2024/11/19-15:21:54.120315 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004951.sst): index type = UintIndex_AllOne , store type = DictZipBlobStore usrkeys = 96773 prefix = 4 raw-key = 0.0008 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0984 GB zip-val = 0.0300 GB avg-val =1017.03 avg-zval = 310.32 2024/11/19-15:21:54.183603 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004951.sst): second pass time = 1.67's, 59.052'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 490.181'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 5.016'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 1.042'MB/sec write SST data time = 0.04's, 774.400'MB/sec dict compress time = 0.00's, 1533.313'MB/sec z-dict build time = 0.00's, sample length = 1.018'MB, throughput = 18249.651'MB/sec zip my value time = 1.67's, unzip length = 0.098'GB zip my value throughput = 59.052'MB/sec zip pipeline throughput = 113.999'MB/sec entries = 96773 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.03 avg-zval = 310.32 usrkeys = 96773 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.03 avg-zval = 310.32 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 4 UnZipSize{ index = 0.0012 GB value = 0.0984 GB dict = 1.02 MB all = 0.1004 GB } __ZipSize{ index = 0.0000 GB value = 0.0300 GB dict = 1.02 MB all = 0.0311 GB } UnZip/Zip{ index =6048.3125 value = 3.2773 dict = 1.00 all = 3.2320 } Zip/UnZip{ index = 0.0002 value = 0.3051 dict = 1.00 all = 0.3094 } ---------------------------- total value len = 8.592444 GB avg = 0.266 KB (by entry num) total key len = 0.828283 GB avg = 0.026 KB total ukey len = 0.570141 GB avg = 0.018 KB total ukey num = 0.032267768 Billion total entry num = 0.032267768 Billion write speed all = 47.768710158 MB/sec (with seq num) write speed all = 46.459775349 MB/sec (without seq num) 2024/11/19-15:21:54.191130 139851179493120 [dcompact_etcd.cc:1163] job-00166/att-00: init time wrt = 0.003, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.326 sec, input = 44.929 MiB 2024/11/19-15:21:54.231531 139851169007360 [dcompact_etcd.cc:1163] job-00167/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.312 sec, input = 127.820 MiB 2024/11/19-15:21:54.322973 139851127064320 [compaction_job.cc:1948] [default] [JOB 156] Generated table #4951: 96773 keys, 31050622 bytes, temperature: kUnknown 2024/11/19-15:21:54.323026 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000914323002, "cf_name": "default", "job": 156, "event": "table_file_creation", "file_number": 4951, "file_size": 31050622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 30030800, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1935460, "raw_average_key_size": 20, "raw_value_size": 98420678, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 96773, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-DictZipBlobStore", "creation_time": 1732000912, "oldest_key_time": 0, "file_creation_time": 1732000912, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4951, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:54.508530 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004949.sst): old prealloc_size = 147639500, real_size = 36359360 2024/11/19-15:21:54.508983 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004949.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926909 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.80 avg-zval = 37.07 2024/11/19-15:21:54.509051 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004949.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 2.945 sec, Key+Value = 0.134 GB 2024/11/19-15:21:54.673593 139851127064320 [compaction_job.cc:703] job-00156: subcompact[0], size: 0.051471 G, files: /storage/mytopling/.rocksdb [004948,004950,004951] 2024/11/19-15:21:54.678993 139851127064320 (Original Log Time 2024/11/19-15:21:50.503667) EVENT_LOG_v1 {"time_micros": 1732000910503648, "job": 156, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4769, 4602, 4603, 4606], "score": 1.52202, "input_data_size": 145850606, "oldest_snapshot_seqno": 131139990} 2024/11/19-15:21:54.679015 139851127064320 (Original Log Time 2024/11/19-15:21:54.677892) [compaction_job.cc:2022] [default] [JOB 156] Compacted 4@2 files to L3 => 51471087 bytes 2024/11/19-15:21:54.679029 139851127064320 (Original Log Time 2024/11/19-15:21:54.678704) [compaction_job.cc:1224] [default] compacted to: files[2 116 46 32 0 0 0] max score 5.97 (141 files need compaction), MB/sec: 35.0 rd, 12.3 wr, level 3, files in(4, 0) out(3 +0 blob) MB in(139.1, 0.0 +0.0 blob) out(49.1 +0.0 blob), read-write-amplify(1.4) write-amplify(0.4) OK, records in: 531853, records dropped: 0 output_compression: Snappy 2024/11/19-15:21:54.679065 139851127064320 (Original Log Time 2024/11/19-15:21:54.678752) EVENT_LOG_v1 {"time_micros": 1732000914678729, "cf": "default", "job": 156, "event": "compaction_finished", "compaction_time_micros": 4169953, "compaction_time_cpu_micros": 1563849, "output_level": 3, "num_output_files": 3, "total_output_size": 51471087, "num_input_records": 531853, "num_output_records": 531853, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 116, 46, 32, 0, 0, 0]} 2024/11/19-15:21:54.679299 139851127064320 [compaction_job.cc:2437] [default] [JOB 168] Compacting 4@1 files to L2, score 5.97, subcompactions 1 : 1 2024/11/19-15:21:54.679322 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 166 Base level 1, inputs: [4857(31MB) 4861(31MB) 4865(31MB) 4869(31MB)] 2024/11/19-15:21:54.985218 139851127064320 [dcompact_etcd.cc:1163] job-00168/att-00: init time wrt = 0.001, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.305 sec, input = 127.822 MiB 2024/11/19-15:21:55.066306 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004949.sst): second pass time = 2.94's, 36.868'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.56's, 66.726'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 4070.482'MB/sec (index lex order gen) rebuild zvType time = 0.46's, 0.501'MB/sec write SST data time = 0.09's, 392.853'MB/sec dict compress time = 0.00's, 1579.341'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 16049.992'MB/sec zip my value time = 2.94's, unzip length = 0.108'GB zip my value throughput = 36.868'MB/sec zip pipeline throughput = 70.385'MB/sec entries = 926909 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 37.07 usrkeys = 926909 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 37.07 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 10 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5120 value = 3.1505 dict = 1.00 all = 3.6063 } Zip/UnZip{ index = 0.0951 value = 0.3174 dict = 1.00 all = 0.2773 } ---------------------------- total value len = 8.700708 GB avg = 0.262 KB (by entry num) total key len = 0.854237 GB avg = 0.026 KB total ukey len = 0.588679 GB avg = 0.018 KB total ukey num = 0.033194677 Billion total entry num = 0.033194677 Billion write speed all = 48.233391029 MB/sec (with seq num) write speed all = 46.892856351 MB/sec (without seq num) 2024/11/19-15:21:55.067546 139851158521600 [compaction_job.cc:1948] [default] [JOB 155] Generated table #4949: 926909 keys, 37217173 bytes, temperature: kUnknown 2024/11/19-15:21:55.067600 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000915067576, "cf_name": "default", "job": 155, "event": "table_file_creation", "file_number": 4949, "file_size": 37217173, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34363952, "index_size": 1763520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953452, "raw_average_key_size": 28, "raw_value_size": 108264410, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926909, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000910, "oldest_key_time": 0, "file_creation_time": 1732000910, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4949, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:55.086254 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004952.sst) /storage/mytopling/..//tmp/Topling-82038-CmoHOr 2024/11/19-15:21:55.106361 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-02 2024/11/19-15:21:55.106442 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00162/att-02/compact.done) = No such file or directory: 7.212 sec on http probe failed: 3, retry 2024/11/19-15:21:55.250756 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-02 2024/11/19-15:21:55.250891 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00164/att-02/compact.done) = No such file or directory: 2.008 sec on http probe failed: 1, retry 2024/11/19-15:21:55.321618 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-00 2024/11/19-15:21:55.321735 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00166/att-00/compact.done) = No such file or directory: 1.123 sec on http probe failed: 1, retry 2024/11/19-15:21:55.416408 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004952.sst): first pass time = 0.33's, 405.390'MB/sec 2024/11/19-15:21:55.416413 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004952.sst: sumWaitingMem = 0.012 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0120 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:21:55.416446 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004952.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.012 GB, dictZip workingMem = 0.0065 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:21:55.488244 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004952.sst): index pass time = 0.07's, 205.961'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 924156 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:21:55.736945 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-02 2024/11/19-15:21:55.737024 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00163/att-02/compact.done) = No such file or directory: 5.198 sec on http probe failed: 2, retry 2024/11/19-15:21:56.236702 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-00 2024/11/19-15:21:56.236822 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00166/att-00/compact.done) = No such file or directory: 2.033 sec on http probe failed: 2, retry 2024/11/19-15:21:56.866763 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-02 2024/11/19-15:21:56.866843 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00164/att-02/compact.done) = No such file or directory: 3.627 sec on http probe failed: 2, retry 2024/11/19-15:21:56.960200 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004952.sst): old prealloc_size = 147639500, real_size = 36258704 2024/11/19-15:21:56.960631 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004952.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924156 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1080 GB zip-val = 0.0343 GB avg-val = 116.82 avg-zval = 37.08 2024/11/19-15:21:56.960700 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004952.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 1.544 sec, Key+Value = 0.134 GB 2024/11/19-15:21:57.111549 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-02 2024/11/19-15:21:57.111647 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00162/att-02/compact.done) = No such file or directory: 9.218 sec on http probe failed: 4, retry 2024/11/19-15:21:57.140899 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-00 2024/11/19-15:21:57.140971 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00166/att-00/compact.done) = No such file or directory: 2.944 sec on http probe failed: 3, retry 2024/11/19-15:21:57.435599 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-00 2024/11/19-15:21:57.435706 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00167/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:21:57.493367 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004952.sst): second pass time = 1.54's, 69.934'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.53's, 69.614'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 4268.365'MB/sec (index lex order gen) rebuild zvType time = 0.44's, 0.529'MB/sec write SST data time = 0.10's, 384.984'MB/sec dict compress time = 0.00's, 1930.978'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 17901.594'MB/sec zip my value time = 1.54's, unzip length = 0.108'GB zip my value throughput = 69.934'MB/sec zip pipeline throughput = 69.934'MB/sec entries = 924156 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.82 avg-zval = 37.08 usrkeys = 924156 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.82 avg-zval = 37.08 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1080 GB dict = 1.08 MB all = 0.1338 GB } __ZipSize{ index = 0.0018 GB value = 0.0343 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5106 value = 3.1503 dict = 1.00 all = 3.6059 } Zip/UnZip{ index = 0.0951 value = 0.3174 dict = 1.00 all = 0.2773 } ---------------------------- total value len = 8.808666 GB avg = 0.258 KB (by entry num) total key len = 0.880113 GB avg = 0.026 KB total ukey len = 0.607162 GB avg = 0.018 KB total ukey num = 0.034118833 Billion total entry num = 0.034118833 Billion write speed all = 48.317014682 MB/sec (with seq num) write speed all = 46.955835789 MB/sec (without seq num) 2024/11/19-15:21:57.494494 139851158521600 [compaction_job.cc:1948] [default] [JOB 155] Generated table #4952: 924156 keys, 37115155 bytes, temperature: kUnknown 2024/11/19-15:21:57.494546 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917494523, "cf_name": "default", "job": 155, "event": "table_file_creation", "file_number": 4952, "file_size": 37115155, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34268928, "index_size": 1758528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25876368, "raw_average_key_size": 28, "raw_value_size": 107957780, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 924156, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732000915, "oldest_key_time": 0, "file_creation_time": 1732000915, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4952, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:57.515501 139851158521600 [compaction_job.cc:703] job-00155: subcompact[0], size: 0.074332 G, files: /storage/mytopling/.rocksdb [004949,004952] 2024/11/19-15:21:57.518052 139851158521600 (Original Log Time 2024/11/19-15:21:50.880779) EVENT_LOG_v1 {"time_micros": 1732000910880752, "job": 155, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4779, 4780, 4781, 4782], "score": 1.60589, "input_data_size": 221689376, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:57.518063 139851158521600 (Original Log Time 2024/11/19-15:21:57.517547) [compaction_job.cc:2022] [default] [JOB 155] Compacted 4@2 files to L3 => 74332328 bytes 2024/11/19-15:21:57.518067 139851158521600 (Original Log Time 2024/11/19-15:21:57.517864) [compaction_job.cc:1224] [default] compacted to: files[2 116 42 34 0 0 0] max score 5.72 (137 files need compaction), MB/sec: 33.4 rd, 11.2 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(211.4, 0.0 +0.0 blob) out(70.9 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 1851065, records dropped: 0 output_compression: Snappy 2024/11/19-15:21:57.518072 139851158521600 (Original Log Time 2024/11/19-15:21:57.517891) EVENT_LOG_v1 {"time_micros": 1732000917517876, "cf": "default", "job": 155, "event": "compaction_finished", "compaction_time_micros": 6634738, "compaction_time_cpu_micros": 4422890, "output_level": 3, "num_output_files": 2, "total_output_size": 74332328, "num_input_records": 1851065, "num_output_records": 1851065, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 116, 42, 34, 0, 0, 0]} 2024/11/19-15:21:57.522517 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004707.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.522543 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917522540, "job": 155, "event": "table_file_deletion", "file_number": 4707} 2024/11/19-15:21:57.526782 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004708.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.526805 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917526802, "job": 155, "event": "table_file_deletion", "file_number": 4708} 2024/11/19-15:21:57.530842 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004709.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.530862 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917530860, "job": 155, "event": "table_file_deletion", "file_number": 4709} 2024/11/19-15:21:57.535024 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004710.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.535045 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917535043, "job": 155, "event": "table_file_deletion", "file_number": 4710} 2024/11/19-15:21:57.539276 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004711.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.539296 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917539294, "job": 155, "event": "table_file_deletion", "file_number": 4711} 2024/11/19-15:21:57.543429 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004712.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.543449 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917543447, "job": 155, "event": "table_file_deletion", "file_number": 4712} 2024/11/19-15:21:57.547612 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004713.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.547631 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917547629, "job": 155, "event": "table_file_deletion", "file_number": 4713} 2024/11/19-15:21:57.551870 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004714.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.551888 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917551886, "job": 155, "event": "table_file_deletion", "file_number": 4714} 2024/11/19-15:21:57.556020 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004715.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.556039 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917556037, "job": 155, "event": "table_file_deletion", "file_number": 4715} 2024/11/19-15:21:57.560438 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004716.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.560457 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917560455, "job": 155, "event": "table_file_deletion", "file_number": 4716} 2024/11/19-15:21:57.564790 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004717.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.564808 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917564807, "job": 155, "event": "table_file_deletion", "file_number": 4717} 2024/11/19-15:21:57.569079 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004718.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.569097 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917569095, "job": 155, "event": "table_file_deletion", "file_number": 4718} 2024/11/19-15:21:57.573456 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004719.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.573475 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917573473, "job": 155, "event": "table_file_deletion", "file_number": 4719} 2024/11/19-15:21:57.577523 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004720.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.577541 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917577539, "job": 155, "event": "table_file_deletion", "file_number": 4720} 2024/11/19-15:21:57.581776 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004721.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.581795 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917581793, "job": 155, "event": "table_file_deletion", "file_number": 4721} 2024/11/19-15:21:57.585907 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004722.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.585933 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917585931, "job": 155, "event": "table_file_deletion", "file_number": 4722} 2024/11/19-15:21:57.590251 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004723.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.590270 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917590268, "job": 155, "event": "table_file_deletion", "file_number": 4723} 2024/11/19-15:21:57.594388 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004724.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.594406 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917594404, "job": 155, "event": "table_file_deletion", "file_number": 4724} 2024/11/19-15:21:57.598566 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004725.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.598585 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917598583, "job": 155, "event": "table_file_deletion", "file_number": 4725} 2024/11/19-15:21:57.602852 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004726.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.602871 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917602869, "job": 155, "event": "table_file_deletion", "file_number": 4726} 2024/11/19-15:21:57.607049 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004727.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.607068 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917607066, "job": 155, "event": "table_file_deletion", "file_number": 4727} 2024/11/19-15:21:57.611178 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004728.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.611196 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917611194, "job": 155, "event": "table_file_deletion", "file_number": 4728} 2024/11/19-15:21:57.615569 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004729.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.615588 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917615586, "job": 155, "event": "table_file_deletion", "file_number": 4729} 2024/11/19-15:21:57.619838 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004730.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.619857 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917619855, "job": 155, "event": "table_file_deletion", "file_number": 4730} 2024/11/19-15:21:57.624060 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004731.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.624084 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917624082, "job": 155, "event": "table_file_deletion", "file_number": 4731} 2024/11/19-15:21:57.628016 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004732.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.628033 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917628032, "job": 155, "event": "table_file_deletion", "file_number": 4732} 2024/11/19-15:21:57.632106 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004733.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.632124 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917632122, "job": 155, "event": "table_file_deletion", "file_number": 4733} 2024/11/19-15:21:57.636167 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004734.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.636185 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917636183, "job": 155, "event": "table_file_deletion", "file_number": 4734} 2024/11/19-15:21:57.640261 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004735.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.640279 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917640277, "job": 155, "event": "table_file_deletion", "file_number": 4735} 2024/11/19-15:21:57.644353 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004736.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.644372 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917644370, "job": 155, "event": "table_file_deletion", "file_number": 4736} 2024/11/19-15:21:57.648287 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004737.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.648305 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917648304, "job": 155, "event": "table_file_deletion", "file_number": 4737} 2024/11/19-15:21:57.652430 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004738.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.652448 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917652447, "job": 155, "event": "table_file_deletion", "file_number": 4738} 2024/11/19-15:21:57.656523 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004739.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.656541 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917656539, "job": 155, "event": "table_file_deletion", "file_number": 4739} 2024/11/19-15:21:57.660490 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004740.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.660509 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917660507, "job": 155, "event": "table_file_deletion", "file_number": 4740} 2024/11/19-15:21:57.663579 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004741.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.663598 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917663596, "job": 155, "event": "table_file_deletion", "file_number": 4741} 2024/11/19-15:21:57.735697 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004746.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.735739 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917735735, "job": 155, "event": "table_file_deletion", "file_number": 4746} 2024/11/19-15:21:57.747121 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-02 2024/11/19-15:21:57.747223 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00163/att-02/compact.done) = No such file or directory: 7.207 sec on http probe failed: 3, retry 2024/11/19-15:21:57.806385 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004759.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.806426 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917806421, "job": 155, "event": "table_file_deletion", "file_number": 4759} 2024/11/19-15:21:57.881575 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004768.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.881664 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917881659, "job": 155, "event": "table_file_deletion", "file_number": 4768} 2024/11/19-15:21:57.952510 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004777.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:57.952553 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000917952549, "job": 155, "event": "table_file_deletion", "file_number": 4777} 2024/11/19-15:21:58.019787 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004786.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:58.019829 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000918019826, "job": 155, "event": "table_file_deletion", "file_number": 4786} 2024/11/19-15:21:58.044454 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-00 2024/11/19-15:21:58.044545 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00166/att-00/compact.done) = No such file or directory: 3.848 sec on http probe failed: 4, retry 2024/11/19-15:21:58.088357 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004799.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:58.088403 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000918088398, "job": 155, "event": "table_file_deletion", "file_number": 4799} 2024/11/19-15:21:58.088592 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004813.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:58.088603 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000918088601, "job": 155, "event": "table_file_deletion", "file_number": 4813} 2024/11/19-15:21:58.088722 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004816.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:58.088730 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000918088729, "job": 155, "event": "table_file_deletion", "file_number": 4816} 2024/11/19-15:21:58.088940 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004819.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:58.088949 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000918088948, "job": 155, "event": "table_file_deletion", "file_number": 4819} 2024/11/19-15:21:58.089163 139851158521600 [compaction_job.cc:2437] [default] [JOB 169] Compacting 4@1 files to L2, score 5.72, subcompactions 1 : 1 2024/11/19-15:21:58.089174 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 167 Base level 1, inputs: [4872(31MB) 4874(31MB) 4876(31MB) 4880(31MB)] 2024/11/19-15:21:58.190716 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-00 2024/11/19-15:21:58.190831 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00168/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:21:58.391273 139851158521600 [dcompact_etcd.cc:1163] job-00169/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.302 sec, input = 127.820 MiB 2024/11/19-15:21:58.482656 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-02 2024/11/19-15:21:58.482764 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00164/att-02/compact.done) = No such file or directory: 5.243 sec on http probe failed: 3, retry 2024/11/19-15:21:58.952366 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-00 2024/11/19-15:21:58.952472 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00166/att-00/compact.done) = No such file or directory: 4.752 sec on http probe failed: 5, giveup 2024/11/19-15:21:58.952491 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00166/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 44.929 sec, real 5.087 (submit 325.510 ms wait 4.752 s), input 44.929 MiB, send shutdown 2024/11/19-15:21:58.957001 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-00 2024/11/19-15:21:58.957261 139851179493120 [dcompact_etcd.cc:1773] job-00166/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:59.121303 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-02 2024/11/19-15:21:59.121435 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00162/att-02/compact.done) = No such file or directory: 11.224 sec on http probe failed: 5, giveup 2024/11/19-15:21:59.121460 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00162/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.238 sec, real 11.545 (submit 311.742 ms wait 11.224 s), input 127.238 MiB, send shutdown 2024/11/19-15:21:59.171423 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00162/att-02 2024/11/19-15:21:59.172088 139851148035840 [dcompact_etcd.cc:1773] job-00162/att-02: CleanFiles 0.001 sec : 0.000 2024/11/19-15:21:59.172370 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00162) = success 2024/11/19-15:21:59.172381 139851148035840 [dcompact_etcd.cc:1773] job-00162/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:21:59.172408 139851148035840 (Original Log Time 2024/11/19-15:21:24.482208) EVENT_LOG_v1 {"time_micros": 1732000884482203, "job": 162, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4820, 4822, 4824, 4827], "score": 6.97131, "input_data_size": 134026816, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:59.172418 139851148035840 [compaction_job.cc:2437] [default] [JOB 162] Compacting 4@1 files to L2, score 6.97, subcompactions 1 : 1 2024/11/19-15:21:59.172431 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 160 Base level 1, inputs: [4820(31MB) 4822(31MB) 4824(31MB) 4827(31MB)] 2024/11/19-15:21:59.172746 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004953.sst) /storage/mytopling/..//tmp/Topling-82038-znczLy 2024/11/19-15:21:59.231714 139851179493120 [dcompact_etcd.cc:1163] job-00166/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.274 sec, input = 44.929 MiB 2024/11/19-15:21:59.243475 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004953.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:21:59.248798 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004953.sst): index pass time = 0.01's, 226.934'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 100306 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:21:59.316357 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004953.sst): first pass time = 0.14's, 467.358'MB/sec 2024/11/19-15:21:59.316413 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004953.sst): old prealloc_size = 73819750, real_size = 65223456 2024/11/19-15:21:59.316485 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004953.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 100306 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0650 GB avg-val = 645.05 avg-zval = 648.30 2024/11/19-15:21:59.316540 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004953.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.073 sec, Key+Value = 0.067 GB 2024/11/19-15:21:59.389134 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004953.sst): second pass time = 0.00's,196066330.303'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 896.773'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2506.402'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.534'MB/sec write SST data time = 0.03's, 2536.425'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 196066330.303'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 100306 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.05 avg-zval = 648.30 usrkeys = 100306 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.05 avg-zval = 648.30 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7943 UnZipSize{ index = 0.0016 GB value = 0.0647 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0002 GB value = 0.0650 GB dict = 0.00 MB all = 0.0652 GB } UnZip/Zip{ index = 9.4343 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 8.873368 GB avg = 0.259 KB (by entry num) total key len = 0.882520 GB avg = 0.026 KB total ukey len = 0.608767 GB avg = 0.018 KB total ukey num = 0.034219139 Billion total entry num = 0.034219139 Billion write speed all = 48.196036673 MB/sec (with seq num) write speed all = 46.843641591 MB/sec (without seq num) 2024/11/19-15:21:59.393500 139851148035840 [compaction_job.cc:1948] [default] [JOB 162] Generated table #4953: 100306 keys, 65200333 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:59.393553 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000919393529, "cf_name": "default", "job": 162, "event": "table_file_creation", "file_number": 4953, "file_size": 65200333, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 65028112, "index_size": 170112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2407344, "raw_average_key_size": 24, "raw_value_size": 64701889, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 100306, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000919, "oldest_key_time": 0, "file_creation_time": 1732000919, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4953, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:59.410084 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004954.sst) /storage/mytopling/..//tmp/Topling-82038-DuFa0S 2024/11/19-15:21:59.440761 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-00 2024/11/19-15:21:59.440834 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00167/att-00/compact.done) = No such file or directory: 5.204 sec on http probe failed: 2, retry 2024/11/19-15:21:59.470025 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004954.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:21:59.475232 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004954.sst): index pass time = 0.01's, 229.792'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 99076 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:21:59.542300 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004954.sst): first pass time = 0.13's, 501.599'MB/sec 2024/11/19-15:21:59.542356 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004954.sst): old prealloc_size = 73819750, real_size = 64446816 2024/11/19-15:21:59.542426 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004954.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 99076 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0639 GB zip-val = 0.0643 GB avg-val = 645.28 avg-zval = 648.53 2024/11/19-15:21:59.542480 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004954.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.072 sec, Key+Value = 0.066 GB 2024/11/19-15:21:59.613873 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004954.sst): second pass time = 0.00's,209611859.016'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 901.010'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2514.819'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.534'MB/sec write SST data time = 0.03's, 2569.538'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.064'GB zip my value throughput = 209611859.016'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 99076 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.28 avg-zval = 648.53 usrkeys = 99076 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.28 avg-zval = 648.53 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7848 UnZipSize{ index = 0.0016 GB value = 0.0639 GB dict = 0.00 MB all = 0.0663 GB } __ZipSize{ index = 0.0002 GB value = 0.0643 GB dict = 0.00 MB all = 0.0644 GB } UnZip/Zip{ index = 9.4358 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 8.937299 GB avg = 0.260 KB (by entry num) total key len = 0.884898 GB avg = 0.026 KB total ukey len = 0.610353 GB avg = 0.018 KB total ukey num = 0.034318215 Billion total entry num = 0.034318215 Billion write speed all = 48.469804284 MB/sec (with seq num) write speed all = 47.114997754 MB/sec (without seq num) 2024/11/19-15:21:59.618828 139851148035840 [compaction_job.cc:1948] [default] [JOB 162] Generated table #4954: 99076 keys, 64423949 bytes (need compaction), temperature: kUnknown 2024/11/19-15:21:59.618883 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000919618858, "cf_name": "default", "job": 162, "event": "table_file_creation", "file_number": 4954, "file_size": 64423949, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64253840, "index_size": 168000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2377824, "raw_average_key_size": 24, "raw_value_size": 63931617, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 99076, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000919, "oldest_key_time": 0, "file_creation_time": 1732000919, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4954, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:21:59.634746 139851148035840 [compaction_job.cc:703] job-00162: subcompact[0], size: 0.129624 G, files: /storage/mytopling/.rocksdb [004953,004954] 2024/11/19-15:21:59.635690 139851148035840 (Original Log Time 2024/11/19-15:21:59.172471) EVENT_LOG_v1 {"time_micros": 1732000919172449, "job": 162, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4820, 4822, 4824, 4827], "score": 6.97131, "input_data_size": 134026816, "oldest_snapshot_seqno": -1} 2024/11/19-15:21:59.635700 139851148035840 (Original Log Time 2024/11/19-15:21:59.635261) [compaction_job.cc:2022] [default] [JOB 162] Compacted 4@1 files to L2 => 129624282 bytes 2024/11/19-15:21:59.635704 139851148035840 (Original Log Time 2024/11/19-15:21:59.635569) [compaction_job.cc:1224] [default] compacted to: files[2 112 44 34 0 0 0] max score 5.47 (135 files need compaction), MB/sec: 289.9 rd, 280.4 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.8, 0.0 +0.0 blob) out(123.6 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 199382, records dropped: 0 output_compression: Snappy 2024/11/19-15:21:59.635720 139851148035840 (Original Log Time 2024/11/19-15:21:59.635594) EVENT_LOG_v1 {"time_micros": 1732000919635582, "cf": "default", "job": 162, "event": "compaction_finished", "compaction_time_micros": 462290, "compaction_time_cpu_micros": 460156, "output_level": 2, "num_output_files": 2, "total_output_size": 129624282, "num_input_records": 199382, "num_output_records": 199382, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 112, 44, 34, 0, 0, 0]} 2024/11/19-15:21:59.641891 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004599.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:21:59.641928 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000919641916, "job": 162, "event": "table_file_deletion", "file_number": 4599} 2024/11/19-15:21:59.642085 139851148035840 [compaction_job.cc:2437] [default] [JOB 170] Compacting 4@1 files to L2, score 5.47, subcompactions 1 : 1 2024/11/19-15:21:59.642094 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 168 Base level 1, inputs: [4882(31MB) 4886(31MB) 4890(31MB) 4893(31MB)] 2024/11/19-15:21:59.753455 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-02 2024/11/19-15:21:59.753533 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00163/att-02/compact.done) = No such file or directory: 9.217 sec on http probe failed: 4, retry 2024/11/19-15:21:59.944712 139851148035840 [dcompact_etcd.cc:1163] job-00170/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.302 sec, input = 127.820 MiB 2024/11/19-15:22:00.099200 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-02 2024/11/19-15:22:00.099273 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00164/att-02/compact.done) = No such file or directory: 6.859 sec on http probe failed: 4, retry 2024/11/19-15:22:00.199676 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-00 2024/11/19-15:22:00.199780 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00168/att-00/compact.done) = No such file or directory: 5.206 sec on http probe failed: 2, retry 2024/11/19-15:22:00.363267 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-01 2024/11/19-15:22:00.363353 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00166/att-01/compact.done) = No such file or directory: 1.123 sec on http probe failed: 1, retry 2024/11/19-15:22:01.270812 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-01 2024/11/19-15:22:01.270904 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00166/att-01/compact.done) = No such file or directory: 2.030 sec on http probe failed: 2, retry 2024/11/19-15:22:01.449285 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-00 2024/11/19-15:22:01.449386 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00167/att-00/compact.done) = No such file or directory: 7.209 sec on http probe failed: 3, retry 2024/11/19-15:22:01.594350 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-00 2024/11/19-15:22:01.594464 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00169/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:22:01.714696 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-02 2024/11/19-15:22:01.714773 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00164/att-02/compact.done) = No such file or directory: 8.476 sec on http probe failed: 5, giveup 2024/11/19-15:22:01.714791 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00164/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 80.332 sec, real 8.810 (submit 325.316 ms wait 8.476 s), input 80.332 MiB, send shutdown 2024/11/19-15:22:01.723297 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00164/att-02 2024/11/19-15:22:01.723533 139851137550080 [dcompact_etcd.cc:1773] job-00164/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:01.725484 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00164) = success 2024/11/19-15:22:01.725494 139851137550080 [dcompact_etcd.cc:1773] job-00164/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:22:01.725516 139851137550080 (Original Log Time 2024/11/19-15:21:35.353032) EVENT_LOG_v1 {"time_micros": 1732000895353025, "job": 164, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4838, 4840, 4843, 4844], "score": 6.47202, "input_data_size": 85591294, "oldest_snapshot_seqno": 132604027} 2024/11/19-15:22:01.725524 139851137550080 [compaction_job.cc:2437] [default] [JOB 164] Compacting 4@1 files to L2, score 6.47, subcompactions 1 : 1 2024/11/19-15:22:01.725532 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 162 Base level 1, inputs: [4838(31MB) 4840(31MB) 4843(1659KB) 4844(16MB)] 2024/11/19-15:22:01.725756 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004955.sst) /storage/mytopling/..//tmp/Topling-82038-fgEPKn 2024/11/19-15:22:01.760980 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-02 2024/11/19-15:22:01.761084 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00163/att-02/compact.done) = No such file or directory: 11.224 sec on http probe failed: 5, giveup 2024/11/19-15:22:01.761102 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00163/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.240 sec, real 11.511 (submit 279.472 ms wait 11.224 s), input 127.240 MiB, send shutdown 2024/11/19-15:22:01.765726 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00163/att-02 2024/11/19-15:22:01.766044 139850978166528 [dcompact_etcd.cc:1773] job-00163/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:01.766336 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00163) = success 2024/11/19-15:22:01.766342 139850978166528 [dcompact_etcd.cc:1773] job-00163/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:01.766361 139850978166528 (Original Log Time 2024/11/19-15:21:27.179007) EVENT_LOG_v1 {"time_micros": 1732000887179001, "job": 163, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4829, 4833, 4834, 4836], "score": 6.72166, "input_data_size": 134027072, "oldest_snapshot_seqno": 131977828} 2024/11/19-15:22:01.766369 139850978166528 [compaction_job.cc:2437] [default] [JOB 163] Compacting 4@1 files to L2, score 6.72, subcompactions 1 : 1 2024/11/19-15:22:01.766377 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 161 Base level 1, inputs: [4829(31MB) 4833(31MB) 4834(31MB) 4836(31MB)] 2024/11/19-15:22:01.766599 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004956.sst) /storage/mytopling/..//tmp/Topling-82038-bCnIGG 2024/11/19-15:22:01.788781 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004955.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:22:01.794602 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004955.sst): index pass time = 0.01's, 207.851'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 100308 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:22:01.827977 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004956.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:22:01.836422 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004956.sst): index pass time = 0.01's, 143.241'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 100300 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:22:01.858098 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004955.sst): first pass time = 0.13's, 507.128'MB/sec 2024/11/19-15:22:01.858146 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004955.sst): old prealloc_size = 73819750, real_size = 65223040 2024/11/19-15:22:01.858217 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004955.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 100308 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0650 GB avg-val = 645.03 avg-zval = 648.28 2024/11/19-15:22:01.858311 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004955.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.070 sec, Key+Value = 0.067 GB 2024/11/19-15:22:01.903343 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004956.sst): first pass time = 0.14's, 490.815'MB/sec 2024/11/19-15:22:01.903401 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004956.sst): old prealloc_size = 73819750, real_size = 65223552 2024/11/19-15:22:01.903475 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004956.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 100300 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0650 GB avg-val = 645.08 avg-zval = 648.34 2024/11/19-15:22:01.903531 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004956.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.076 sec, Key+Value = 0.067 GB 2024/11/19-15:22:01.927765 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004955.sst): second pass time = 0.00's,212136022.951'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 940.557'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2520.327'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.554'MB/sec write SST data time = 0.02's, 2698.108'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 212136022.951'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 100308 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.03 avg-zval = 648.28 usrkeys = 100308 avg-key = 16.00 avg-zkey = 1.70 avg-val = 652.03 avg-zval = 648.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 100308 bytes = 263328 } seq expand size = 702156 multi value expand size = 0 cnt WriteAppend = 7946 UnZipSize{ index = 0.0016 GB value = 0.0647 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0002 GB value = 0.0650 GB dict = 0.00 MB all = 0.0655 GB } UnZip/Zip{ index = 9.4345 value = 0.9950 dict = 0.00 all = 1.0251 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9755 } ---------------------------- total value len = 9.002703 GB avg = 0.262 KB (by entry num) total key len = 0.887306 GB avg = 0.026 KB total ukey len = 0.611957 GB avg = 0.018 KB total ukey num = 0.034418523 Billion total entry num = 0.034418523 Billion write speed all = 48.253455573 MB/sec (with seq num) write speed all = 46.910028889 MB/sec (without seq num) 2024/11/19-15:22:01.932072 139851137550080 [compaction_job.cc:1948] [default] [JOB 164] Generated table #4955: 100308 keys, 65463258 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:01.932127 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000921932102, "cf_name": "default", "job": 164, "event": "table_file_creation", "file_number": 4955, "file_size": 65463258, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128250275, "largest_seqno": 130128313, "table_properties": {"data_size": 65027696, "index_size": 170112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2407392, "raw_average_key_size": 24, "raw_value_size": 64701487, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 100308, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000921, "oldest_key_time": 0, "file_creation_time": 1732000921, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4955, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:01.948501 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004957.sst) /storage/mytopling/..//tmp/Topling-82038-tqxqPM 2024/11/19-15:22:01.949852 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004957.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.001 GB 2024/11/19-15:22:01.949986 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004957.sst): index pass time = 0.00's, 221.517'MB/sec index type = UintIndex_AllOne usrkeys = 1920 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 10 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 12.00 avg-zkey = 0.09 2024/11/19-15:22:01.951285 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004957.sst): first pass time = 0.00's, 463.814'MB/sec 2024/11/19-15:22:01.951323 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004957.sst): old prealloc_size = 73819750, real_size = 1243016 2024/11/19-15:22:01.951343 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004957.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 1920 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 12.00 avg-zkey = 0.09 raw-val = 0.0012 GB zip-val = 0.0012 GB avg-val = 644.24 avg-zval = 646.98 2024/11/19-15:22:01.952856 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004957.sst): second pass time = 0.00's,4497949.091'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 821.397'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.704'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.466'MB/sec write SST data time = 0.00's, 2665.465'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.001'GB zip my value throughput = 4497949.091'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 1920 avg-key = 24.00 avg-zkey = 0.10 avg-val = 644.24 avg-zval = 646.98 usrkeys = 1920 avg-key = 16.00 avg-zkey = 0.10 avg-val = 651.24 avg-zval = 646.98 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 1920 bytes = 2656 } seq expand size = 13440 multi value expand size = 0 cnt WriteAppend = 159 UnZipSize{ index = 0.0000 GB value = 0.0012 GB dict = 0.00 MB all = 0.0013 GB } __ZipSize{ index = 0.0000 GB value = 0.0012 GB dict = 0.00 MB all = 0.0012 GB } UnZip/Zip{ index = 160.0000 value = 0.9958 dict = 0.00 all = 1.0295 } Zip/UnZip{ index = 0.0063 value = 1.0042 dict = 0.00 all = 0.9713 } ---------------------------- total value len = 9.003953 GB avg = 0.262 KB (by entry num) total key len = 0.887352 GB avg = 0.026 KB total ukey len = 0.611988 GB avg = 0.018 KB total ukey num = 0.034420443 Billion total entry num = 0.034420443 Billion write speed all = 48.253871836 MB/sec (with seq num) write speed all = 46.910534716 MB/sec (without seq num) 2024/11/19-15:22:01.953253 139851137550080 [compaction_job.cc:1948] [default] [JOB 164] Generated table #4957: 1920 keys, 1246233 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:01.953297 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000921953275, "cf_name": "default", "job": 164, "event": "table_file_creation", "file_number": 4957, "file_size": 1246233, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130128314, "largest_seqno": 130130233, "table_properties": {"data_size": 1242192, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 46080, "raw_average_key_size": 24, "raw_value_size": 1236936, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 1920, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000921, "oldest_key_time": 0, "file_creation_time": 1732000921, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4957, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:01.953778 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004958.sst) /storage/mytopling/..//tmp/Topling-82038-TxlNQT 2024/11/19-15:22:01.974231 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004956.sst): second pass time = 0.00's,81181938.519'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 920.640'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2424.837'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.529'MB/sec write SST data time = 0.02's, 2796.764'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 81181938.519'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 100300 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.08 avg-zval = 648.34 usrkeys = 100300 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.08 avg-zval = 648.34 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7943 UnZipSize{ index = 0.0016 GB value = 0.0647 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0002 GB value = 0.0650 GB dict = 0.00 MB all = 0.0652 GB } UnZip/Zip{ index = 9.4338 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 9.068655 GB avg = 0.263 KB (by entry num) total key len = 0.889759 GB avg = 0.026 KB total ukey len = 0.613593 GB avg = 0.018 KB total ukey num = 0.034520743 Billion total entry num = 0.034520743 Billion write speed all = 48.576193287 MB/sec (with seq num) write speed all = 47.229082189 MB/sec (without seq num) 2024/11/19-15:22:01.979075 139850978166528 [compaction_job.cc:1948] [default] [JOB 163] Generated table #4956: 100300 keys, 65200397 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:01.979125 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000921979101, "cf_name": "default", "job": 163, "event": "table_file_creation", "file_number": 4956, "file_size": 65200397, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 65028208, "index_size": 170112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2407200, "raw_average_key_size": 24, "raw_value_size": 64702005, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 100300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000921, "oldest_key_time": 0, "file_creation_time": 1732000921, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4956, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:01.995093 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004959.sst) /storage/mytopling/..//tmp/Topling-82038-Xs6Fbl 2024/11/19-15:22:02.012863 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004958.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.016 GB 2024/11/19-15:22:02.012987 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004958.sst): index pass time = 0.00's,18775.227'MB/sec index type = UintIndex_AllOne usrkeys = 214108 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0017 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:22:02.030755 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004958.sst): first pass time = 0.08's, 205.831'MB/sec 2024/11/19-15:22:02.030803 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004958.sst): old prealloc_size = 73819750, real_size = 12256280 2024/11/19-15:22:02.030824 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004958.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 214108 prefix = 4 raw-key = 0.0017 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0116 GB zip-val = 0.0122 GB avg-val = 53.99 avg-zval = 56.99 2024/11/19-15:22:02.044661 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4960. Immutable memtables: 0. 2024/11/19-15:22:02.045410 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 171] Syncing log #4942 2024/11/19-15:22:02.045518 139850967680768 (Original Log Time 2024/11/19-15:22:02.045195) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:22:02.045531 139850967680768 [flush_job.cc:879] [default] [JOB 171] Flushing memtable with next log file: 4960 2024/11/19-15:22:02.045559 139850967680768 EVENT_LOG_v1 {"time_micros": 1732000922045550, "job": 171, "event": "flush_started", "num_memtables": 1, "num_entries": 1659867, "num_deletes": 0, "total_data_size": 475928506, "memory_usage": 541588872, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:22:02.045565 139850967680768 [flush_job.cc:910] [default] [JOB 171] Level-0 flush table #4961: started 2024/11/19-15:22:02.046339 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000020.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004961.sst in ConvertToSST, mem_size = 516.499 M, time = 759.265 us 2024/11/19-15:22:02.062261 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004959.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:22:02.068415 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004958.sst): second pass time = 0.00's,33217985.632'MB/sec, value only(73.0% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.04's, 340.593'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.896'MB/sec (index lex order gen) rebuild zvType time = 0.01's, 4.084'MB/sec write SST data time = 0.02's, 523.539'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.012'GB zip my value throughput = 33217985.632'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 214108 avg-key = 20.00 avg-zkey = 0.00 avg-val = 53.99 avg-zval = 56.99 usrkeys = 214108 avg-key = 12.00 avg-zkey = 0.00 avg-val = 60.99 avg-zval = 56.99 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 214108 bytes = 588816 } seq expand size = 1498756 multi value expand size = 0 cnt WriteAppend = 1497 UnZipSize{ index = 0.0026 GB value = 0.0116 GB dict = 0.00 MB all = 0.0158 GB } __ZipSize{ index = 0.0000 GB value = 0.0122 GB dict = 0.00 MB all = 0.0128 GB } UnZip/Zip{ index =13381.7500 value = 0.9473 dict = 0.00 all = 1.2384 } Zip/UnZip{ index = 0.0001 value = 1.0556 dict = 0.00 all = 0.8075 } ---------------------------- total value len = 9.081714 GB avg = 0.261 KB (by entry num) total key len = 0.894041 GB avg = 0.026 KB total ukey len = 0.616162 GB avg = 0.018 KB total ukey num = 0.034734851 Billion total entry num = 0.034734851 Billion write speed all = 48.638443563 MB/sec (with seq num) write speed all = 47.283599467 MB/sec (without seq num) 2024/11/19-15:22:02.070775 139851137550080 [compaction_job.cc:1948] [default] [JOB 164] Generated table #4958: 214108 keys, 12792597 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:02.070864 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000922070818, "cf_name": "default", "job": 164, "event": "table_file_creation", "file_number": 4958, "file_size": 12792597, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121863421, "largest_seqno": 125337014, "table_properties": {"data_size": 12202400, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4282160, "raw_average_key_size": 20, "raw_value_size": 11559859, "raw_average_value_size": 53, "num_data_blocks": 1, "num_entries": 214108, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000921, "oldest_key_time": 0, "file_creation_time": 1732000921, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4958, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:02.073325 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004959.sst): index pass time = 0.01's, 107.920'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 99090 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:22:02.077089 139851137550080 [compaction_job.cc:703] job-00164: subcompact[0], size: 0.079502 G, files: /storage/mytopling/.rocksdb [004955,004957,004958] 2024/11/19-15:22:02.078782 139851137550080 (Original Log Time 2024/11/19-15:22:01.725562) EVENT_LOG_v1 {"time_micros": 1732000921725545, "job": 164, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4838, 4840, 4843, 4844], "score": 6.47202, "input_data_size": 85591294, "oldest_snapshot_seqno": 132604027} 2024/11/19-15:22:02.078794 139851137550080 (Original Log Time 2024/11/19-15:22:02.078065) [compaction_job.cc:2022] [default] [JOB 164] Compacted 4@1 files to L2 => 79502088 bytes 2024/11/19-15:22:02.078798 139851137550080 (Original Log Time 2024/11/19-15:22:02.078555) [compaction_job.cc:1224] [default] compacted to: files[2 108 47 34 0 0 0] max score 5.22 (134 files need compaction), MB/sec: 243.5 rd, 226.1 wr, level 2, files in(4, 0) out(3 +0 blob) MB in(81.6, 0.0 +0.0 blob) out(75.8 +0.0 blob), read-write-amplify(1.9) write-amplify(0.9) OK, records in: 316336, records dropped: 0 output_compression: Snappy 2024/11/19-15:22:02.078821 139851137550080 (Original Log Time 2024/11/19-15:22:02.078601) EVENT_LOG_v1 {"time_micros": 1732000922078578, "cf": "default", "job": 164, "event": "compaction_finished", "compaction_time_micros": 351548, "compaction_time_cpu_micros": 346257, "output_level": 2, "num_output_files": 3, "total_output_size": 79502088, "num_input_records": 316336, "num_output_records": 316336, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 108, 47, 34, 0, 0, 0]} 2024/11/19-15:22:02.079890 139851137550080 [compaction_job.cc:2437] [default] [JOB 172] Compacting 4@1 files to L2, score 5.22, subcompactions 1 : 1 2024/11/19-15:22:02.079931 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 169 Base level 1, inputs: [4896(31MB) 4899(31MB) 4905(31MB) 4908(31MB)] 2024/11/19-15:22:02.177817 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-01 2024/11/19-15:22:02.177963 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00166/att-01/compact.done) = No such file or directory: 2.938 sec on http probe failed: 3, retry 2024/11/19-15:22:02.205045 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-00 2024/11/19-15:22:02.205178 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00168/att-00/compact.done) = No such file or directory: 7.215 sec on http probe failed: 3, retry 2024/11/19-15:22:02.224849 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004959.sst): first pass time = 0.23's, 288.643'MB/sec 2024/11/19-15:22:02.224940 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004959.sst): old prealloc_size = 73819750, real_size = 64449072 2024/11/19-15:22:02.225067 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004959.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 99090 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0639 GB zip-val = 0.0643 GB avg-val = 645.21 avg-zval = 648.46 2024/11/19-15:22:02.225153 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004959.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.163 sec, Key+Value = 0.066 GB 2024/11/19-15:22:02.267019 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004961.sst): fsize = 516.501 M, time(ms): open: 0.045, seek: 0.019, finish: 0.051, meta: 0.012, Flush: 0.014, sync: 220.504, close: 0.011, all: 220.656 2024/11/19-15:22:02.268189 139850967680768 [flush_job.cc:1083] [default] [JOB 171] Flush lasted 222672 microseconds, and 200417 cpu microseconds. 2024/11/19-15:22:02.289970 139850967680768 (Original Log Time 2024/11/19-15:22:02.267067) [flush_job.cc:1029] [default] [JOB 171] Level-0 flush table #4961: 541590298 bytes OK (needs compaction) 2024/11/19-15:22:02.289981 139850967680768 (Original Log Time 2024/11/19-15:22:02.268205) [memtable_list.cc:560] [default] Level-0 commit table #4961 started 2024/11/19-15:22:02.289996 139850967680768 (Original Log Time 2024/11/19-15:22:02.289885) [memtable_list.cc:759] [default] Level-0 commit table #4961: memtable #1 done 2024/11/19-15:22:02.289999 139850967680768 (Original Log Time 2024/11/19-15:22:02.289911) EVENT_LOG_v1 {"time_micros": 1732000922289906, "job": 171, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [3, 108, 47, 34, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:22:02.290002 139850967680768 (Original Log Time 2024/11/19-15:22:02.289941) [db_impl_compaction_flush.cc:310] [default] Level summary: files[3 108 47 34 0 0 0] max score 4.97 (131 files need compaction) 2024/11/19-15:22:02.312150 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004959.sst): second pass time = 0.00's,136903000.000'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 738.631'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1379.874'MB/sec (index lex order gen) rebuild zvType time = 0.06's, 0.408'MB/sec write SST data time = 0.03's, 2438.787'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.064'GB zip my value throughput = 136903000.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 99090 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.21 avg-zval = 648.46 usrkeys = 99090 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.21 avg-zval = 648.46 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7848 UnZipSize{ index = 0.0016 GB value = 0.0639 GB dict = 0.00 MB all = 0.0663 GB } __ZipSize{ index = 0.0002 GB value = 0.0643 GB dict = 0.00 MB all = 0.0644 GB } UnZip/Zip{ index = 9.4300 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 9.145648 GB avg = 0.263 KB (by entry num) total key len = 0.896419 GB avg = 0.026 KB total ukey len = 0.617748 GB avg = 0.018 KB total ukey num = 0.034833941 Billion total entry num = 0.034833941 Billion write speed all = 48.903633550 MB/sec (with seq num) write speed all = 47.546537395 MB/sec (without seq num) 2024/11/19-15:22:02.316183 139850978166528 [compaction_job.cc:1948] [default] [JOB 163] Generated table #4959: 99090 keys, 64426189 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:02.316227 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000922316205, "cf_name": "default", "job": 163, "event": "table_file_creation", "file_number": 4959, "file_size": 64426189, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64255968, "index_size": 168128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2378160, "raw_average_key_size": 24, "raw_value_size": 63933701, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 99090, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000921, "oldest_key_time": 0, "file_creation_time": 1732000921, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4959, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:02.332600 139850978166528 [compaction_job.cc:703] job-00163: subcompact[0], size: 0.129627 G, files: /storage/mytopling/.rocksdb [004956,004959] 2024/11/19-15:22:02.333499 139850978166528 (Original Log Time 2024/11/19-15:22:01.766408) EVENT_LOG_v1 {"time_micros": 1732000921766389, "job": 163, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4829, 4833, 4834, 4836], "score": 6.72166, "input_data_size": 134027072, "oldest_snapshot_seqno": 131977828} 2024/11/19-15:22:02.333508 139850978166528 (Original Log Time 2024/11/19-15:22:02.333083) [compaction_job.cc:2022] [default] [JOB 163] Compacted 4@1 files to L2 => 129626586 bytes 2024/11/19-15:22:02.333512 139850978166528 (Original Log Time 2024/11/19-15:22:02.333368) [compaction_job.cc:1224] [default] compacted to: files[3 104 49 34 0 0 0] max score 4.97 (133 files need compaction), MB/sec: 236.7 rd, 228.9 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.8, 0.0 +0.0 blob) out(123.6 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 199390, records dropped: 0 output_compression: Snappy 2024/11/19-15:22:02.333530 139850978166528 (Original Log Time 2024/11/19-15:22:02.333392) EVENT_LOG_v1 {"time_micros": 1732000922333379, "cf": "default", "job": 163, "event": "compaction_finished", "compaction_time_micros": 566204, "compaction_time_cpu_micros": 520382, "output_level": 2, "num_output_files": 2, "total_output_size": 129626586, "num_input_records": 199390, "num_output_records": 199390, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 104, 49, 34, 0, 0, 0]} 2024/11/19-15:22:02.343059 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004763.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:02.343084 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000922343081, "job": 163, "event": "table_file_deletion", "file_number": 4763} 2024/11/19-15:22:02.352562 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004764.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:02.352583 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000922352581, "job": 163, "event": "table_file_deletion", "file_number": 4764} 2024/11/19-15:22:02.362598 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004765.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:02.362619 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000922362617, "job": 163, "event": "table_file_deletion", "file_number": 4765} 2024/11/19-15:22:02.371993 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004766.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:02.372014 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000922372012, "job": 163, "event": "table_file_deletion", "file_number": 4766} 2024/11/19-15:22:02.379201 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004772.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:02.379222 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000922379219, "job": 163, "event": "table_file_deletion", "file_number": 4772} 2024/11/19-15:22:02.386007 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004773.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:02.386028 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000922386026, "job": 163, "event": "table_file_deletion", "file_number": 4773} 2024/11/19-15:22:02.390850 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004774.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:02.390869 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000922390867, "job": 163, "event": "table_file_deletion", "file_number": 4774} 2024/11/19-15:22:02.394915 139851137550080 [dcompact_etcd.cc:1163] job-00172/att-00: init time wrt = 0.001, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.314 sec, input = 127.821 MiB 2024/11/19-15:22:02.398158 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004775.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:02.398180 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000922398177, "job": 163, "event": "table_file_deletion", "file_number": 4775} 2024/11/19-15:22:02.398367 139850978166528 [compaction_job.cc:2437] [default] [JOB 173] Compacting 2@1 files to L2, score 4.97, subcompactions 1 : 1 2024/11/19-15:22:02.398376 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 171 Base level 1, inputs: [4912(31MB) 4916(24MB)] 2024/11/19-15:22:02.712050 139850978166528 [dcompact_etcd.cc:1163] job-00173/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.313 sec, input = 56.703 MiB 2024/11/19-15:22:03.085621 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-01 2024/11/19-15:22:03.085719 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00166/att-01/compact.done) = No such file or directory: 3.845 sec on http probe failed: 4, retry 2024/11/19-15:22:03.146480 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-00 2024/11/19-15:22:03.146584 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00170/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:22:03.458821 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-00 2024/11/19-15:22:03.458984 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00167/att-00/compact.done) = No such file or directory: 9.218 sec on http probe failed: 4, retry 2024/11/19-15:22:03.603471 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-00 2024/11/19-15:22:03.603565 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00169/att-00/compact.done) = No such file or directory: 5.203 sec on http probe failed: 2, retry 2024/11/19-15:22:03.989407 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-01 2024/11/19-15:22:03.989487 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00166/att-01/compact.done) = No such file or directory: 4.753 sec on http probe failed: 5, giveup 2024/11/19-15:22:03.989505 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00166/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 44.929 sec, real 5.032 (submit 274.369 ms wait 4.753 s), input 44.929 MiB, send shutdown 2024/11/19-15:22:03.997874 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-01 2024/11/19-15:22:03.998110 139851179493120 [dcompact_etcd.cc:1773] job-00166/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:04.138112 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-00 2024/11/19-15:22:04.138185 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00173/att-00/compact.done) = No such file or directory: 1.418 sec on http probe failed: 1, retry 2024/11/19-15:22:04.210539 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-00 2024/11/19-15:22:04.210612 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00168/att-00/compact.done) = No such file or directory: 9.220 sec on http probe failed: 4, retry 2024/11/19-15:22:04.285565 139851179493120 [dcompact_etcd.cc:1163] job-00166/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.287 sec, input = 44.929 MiB 2024/11/19-15:22:05.155250 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-00 2024/11/19-15:22:05.155353 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00170/att-00/compact.done) = No such file or directory: 5.202 sec on http probe failed: 2, retry 2024/11/19-15:22:05.282305 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-00 2024/11/19-15:22:05.282384 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00173/att-00/compact.done) = No such file or directory: 2.560 sec on http probe failed: 2, retry 2024/11/19-15:22:05.417407 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-02 2024/11/19-15:22:05.417489 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00166/att-02/compact.done) = No such file or directory: 1.123 sec on http probe failed: 1, retry 2024/11/19-15:22:05.466386 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-00 2024/11/19-15:22:05.466449 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00167/att-00/compact.done) = No such file or directory: 11.228 sec on http probe failed: 5, giveup 2024/11/19-15:22:05.466467 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00167/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.546 (submit 311.533 ms wait 11.228 s), input 127.820 MiB, send shutdown 2024/11/19-15:22:05.474629 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-00 2024/11/19-15:22:05.474864 139851169007360 [dcompact_etcd.cc:1773] job-00167/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:05.599838 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-00 2024/11/19-15:22:05.599939 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00172/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:22:05.612236 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-00 2024/11/19-15:22:05.612316 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00169/att-00/compact.done) = No such file or directory: 7.212 sec on http probe failed: 3, retry 2024/11/19-15:22:05.779319 139851169007360 [dcompact_etcd.cc:1163] job-00167/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.304 sec, input = 127.820 MiB 2024/11/19-15:22:06.218262 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-00 2024/11/19-15:22:06.218332 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00168/att-00/compact.done) = No such file or directory: 11.225 sec on http probe failed: 5, giveup 2024/11/19-15:22:06.218351 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00168/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.822 sec, real 11.538 (submit 304.861 ms wait 11.225 s), input 127.822 MiB, send shutdown 2024/11/19-15:22:06.226671 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-00 2024/11/19-15:22:06.226900 139851127064320 [dcompact_etcd.cc:1773] job-00168/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:06.323695 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-02 2024/11/19-15:22:06.323762 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00166/att-02/compact.done) = No such file or directory: 2.031 sec on http probe failed: 2, retry 2024/11/19-15:22:06.424044 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-00 2024/11/19-15:22:06.424118 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00173/att-00/compact.done) = No such file or directory: 3.704 sec on http probe failed: 3, retry 2024/11/19-15:22:06.509542 139851127064320 [dcompact_etcd.cc:1163] job-00168/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.283 sec, input = 127.822 MiB 2024/11/19-15:22:07.160448 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-00 2024/11/19-15:22:07.160522 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00170/att-00/compact.done) = No such file or directory: 7.211 sec on http probe failed: 3, retry 2024/11/19-15:22:07.228555 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-02 2024/11/19-15:22:07.228641 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00166/att-02/compact.done) = No such file or directory: 2.937 sec on http probe failed: 3, retry 2024/11/19-15:22:07.563683 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-00 2024/11/19-15:22:07.563767 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00173/att-00/compact.done) = No such file or directory: 4.846 sec on http probe failed: 4, retry 2024/11/19-15:22:07.607343 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-00 2024/11/19-15:22:07.607409 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00172/att-00/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:22:07.620996 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-00 2024/11/19-15:22:07.621057 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00169/att-00/compact.done) = No such file or directory: 9.221 sec on http probe failed: 4, retry 2024/11/19-15:22:08.132286 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-02 2024/11/19-15:22:08.132387 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00166/att-02/compact.done) = No such file or directory: 3.842 sec on http probe failed: 4, retry 2024/11/19-15:22:08.703692 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-00 2024/11/19-15:22:08.703779 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00173/att-00/compact.done) = No such file or directory: 5.986 sec on http probe failed: 5, giveup 2024/11/19-15:22:08.703798 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00173/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 56.703 sec, real 6.305 (submit 313.246 ms wait 5.986 s), input 56.703 MiB, send shutdown 2024/11/19-15:22:08.709343 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-00 2024/11/19-15:22:08.709555 139850978166528 [dcompact_etcd.cc:1773] job-00173/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:08.964217 139850978166528 [dcompact_etcd.cc:1163] job-00173/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.255 sec, input = 56.703 MiB 2024/11/19-15:22:08.984747 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-01 2024/11/19-15:22:08.984840 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00167/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:22:09.039364 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-02 2024/11/19-15:22:09.039430 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00166/att-02/compact.done) = No such file or directory: 4.745 sec on http probe failed: 5, giveup 2024/11/19-15:22:09.039447 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00166/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 44.929 sec, real 5.041 (submit 287.377 ms wait 4.745 s), input 44.929 MiB, send shutdown 2024/11/19-15:22:09.046985 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00166/att-02 2024/11/19-15:22:09.047203 139851179493120 [dcompact_etcd.cc:1773] job-00166/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:09.049007 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00166) = success 2024/11/19-15:22:09.049017 139851179493120 [dcompact_etcd.cc:1773] job-00166/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:22:09.049043 139851179493120 (Original Log Time 2024/11/19-15:21:53.862835) EVENT_LOG_v1 {"time_micros": 1732000913862821, "job": 166, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4817, 4826, 4831], "score": 6.31259, "input_data_size": 49030420, "oldest_snapshot_seqno": -1} 2024/11/19-15:22:09.049051 139851179493120 [compaction_job.cc:2437] [default] [JOB 166] Compacting 3@1 files to L2, score 6.31, subcompactions 1 : 1 2024/11/19-15:22:09.049059 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 164 Base level 1, inputs: [4817(21MB) 4826(3810KB) 4831(21MB)] 2024/11/19-15:22:09.049286 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004962.sst) /storage/mytopling/..//tmp/Topling-82038-4CY09J 2024/11/19-15:22:09.135936 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004962.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.021 GB 2024/11/19-15:22:09.136070 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004962.sst): index pass time = 0.00's,22585.394'MB/sec index type = UintIndex_AllOne usrkeys = 289892 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0023 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:22:09.162736 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004962.sst): first pass time = 0.11's, 189.097'MB/sec 2024/11/19-15:22:09.162783 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004962.sst): old prealloc_size = 53933462, real_size = 16595448 2024/11/19-15:22:09.162806 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004962.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 289892 prefix = 4 raw-key = 0.0023 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0157 GB zip-val = 0.0165 GB avg-val = 54.00 avg-zval = 57.00 2024/11/19-15:22:09.169644 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-00 2024/11/19-15:22:09.169772 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00170/att-00/compact.done) = No such file or directory: 9.216 sec on http probe failed: 4, retry 2024/11/19-15:22:09.201350 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004962.sst): second pass time = 0.00's,37626742.788'MB/sec, value only(73.0% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.04's, 450.234'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.828'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 4.088'MB/sec write SST data time = 0.02's, 834.308'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.016'GB zip my value throughput = 37626742.788'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 289892 avg-key = 20.00 avg-zkey = 0.00 avg-val = 54.00 avg-zval = 57.00 usrkeys = 289892 avg-key = 12.00 avg-zkey = 0.00 avg-val = 61.00 avg-zval = 57.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 289892 bytes = 833456 } seq expand size = 2029244 multi value expand size = 0 cnt WriteAppend = 2023 UnZipSize{ index = 0.0035 GB value = 0.0157 GB dict = 0.00 MB all = 0.0215 GB } __ZipSize{ index = 0.0000 GB value = 0.0165 GB dict = 0.00 MB all = 0.0174 GB } UnZip/Zip{ index =18118.2500 value = 0.9474 dict = 0.00 all = 1.2358 } Zip/UnZip{ index = 0.0001 value = 1.0556 dict = 0.00 all = 0.8092 } ---------------------------- total value len = 9.163330 GB avg = 0.261 KB (by entry num) total key len = 0.902217 GB avg = 0.026 KB total ukey len = 0.621226 GB avg = 0.018 KB total ukey num = 0.035123833 Billion total entry num = 0.035123833 Billion write speed all = 47.426828404 MB/sec (with seq num) write speed all = 46.102856991 MB/sec (without seq num) 2024/11/19-15:22:09.202816 139851179493120 [compaction_job.cc:1948] [default] [JOB 166] Generated table #4962: 289892 keys, 17357397 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:09.202883 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000929202843, "cf_name": "default", "job": 166, "event": "table_file_creation", "file_number": 4962, "file_size": 17357397, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125337015, "largest_seqno": 130136235, "table_properties": {"data_size": 16522624, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 5797840, "raw_average_key_size": 20, "raw_value_size": 15652725, "raw_average_value_size": 53, "num_data_blocks": 1, "num_entries": 289892, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000929, "oldest_key_time": 0, "file_creation_time": 1732000929, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4962, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:09.207423 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004963.sst) /storage/mytopling/..//tmp/Topling-82038-dYv4EP 2024/11/19-15:22:09.243303 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004963.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0013 GB, waited 0.000 sec, Key+Value = 0.004 GB 2024/11/19-15:22:09.244551 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004963.sst): first pass time = 0.04's, 96.607'MB/sec 2024/11/19-15:22:09.252043 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004963.sst): index pass time = 0.01's, 205.633'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 149400 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0018 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:22:09.252105 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004963.sst): old prealloc_size = 53933462, real_size = 290976 2024/11/19-15:22:09.252183 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004963.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZeroLengthBlobStore usrkeys = 149400 prefix = 4 raw-key = 0.0018 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 0.00 avg-zval = 0.00 2024/11/19-15:22:09.252263 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004963.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.009 sec, Key+Value = 0.004 GB 2024/11/19-15:22:09.264866 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004963.sst): second pass time = 0.00's, 0.000'MB/sec, value only( 0.0% of KV) wait indexing time = 0.01's, remap KeyValue time = 0.01's, 53.746'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3402.436'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 386.290'MB/sec write SST data time = 0.01's, 54.473'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 0.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 149400 avg-key = 24.00 avg-zkey = 1.70 avg-val = 0.00 avg-zval = 0.00 usrkeys = 149400 avg-key = 16.00 avg-zkey = 1.70 avg-val = 7.00 avg-zval = 0.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 149400 bytes = 429536 } seq expand size = 1045800 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0024 GB value = 0.0000 GB dict = 0.00 MB all = 0.0036 GB } __ZipSize{ index = 0.0003 GB value = 0.0000 GB dict = 0.00 MB all = 0.0007 GB } UnZip/Zip{ index = 9.4342 value = 0.0000 dict = 0.00 all = 5.2337 } Zip/UnZip{ index = 0.1060 value = inf dict = 0.00 all = 0.1911 } ---------------------------- total value len = 9.164375 GB avg = 0.260 KB (by entry num) total key len = 0.905803 GB avg = 0.026 KB total ukey len = 0.623617 GB avg = 0.018 KB total ukey num = 0.035273233 Billion total entry num = 0.035273233 Billion write speed all = 47.434455145 MB/sec (with seq num) write speed all = 46.105249976 MB/sec (without seq num) 2024/11/19-15:22:09.265357 139851179493120 [compaction_job.cc:1948] [default] [JOB 166] Generated table #4963: 149400 keys, 685094 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:09.265401 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000929265379, "cf_name": "default", "job": 166, "event": "table_file_creation", "file_number": 4963, "file_size": 685094, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121919568, "largest_seqno": 130124228, "table_properties": {"data_size": 80, "index_size": 253376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3585600, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 149400, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZeroLengthBlobStore", "creation_time": 1732000929, "oldest_key_time": 0, "file_creation_time": 1732000929, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4963, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:09.265967 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004964.sst) /storage/mytopling/..//tmp/Topling-82038-BRUhEx 2024/11/19-15:22:09.392939 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004964.sst: sumWaitingMem = 0.005 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0045 GB, waited 0.000 sec, Key+Value = 0.022 GB 2024/11/19-15:22:09.418647 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004964.sst): index pass time = 0.03's, 235.559'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 504000 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0060 GB zip-key = 0.0008 GB avg-key = 12.00 avg-zkey = 1.68 2024/11/19-15:22:09.423311 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004964.sst): first pass time = 0.16's, 140.310'MB/sec 2024/11/19-15:22:09.423356 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004964.sst): old prealloc_size = 53933462, real_size = 11448672 2024/11/19-15:22:09.423568 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004964.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 504000 prefix = 4 raw-key = 0.0060 GB zip-key = 0.0008 GB avg-key = 12.00 avg-zkey = 1.68 raw-val = 0.0100 GB zip-val = 0.0105 GB avg-val = 19.80 avg-zval = 20.78 2024/11/19-15:22:09.423860 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004964.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0012 GB, waited 0.031 sec, Key+Value = 0.022 GB 2024/11/19-15:22:09.484993 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004964.sst): second pass time = 0.00's,30331914.894'MB/sec, value only(45.2% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 207.337'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 4089.438'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 7.226'MB/sec write SST data time = 0.04's, 290.515'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.010'GB zip my value throughput = 30331914.894'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 504000 avg-key = 24.00 avg-zkey = 1.68 avg-val = 19.80 avg-zval = 20.78 usrkeys = 504000 avg-key = 16.00 avg-zkey = 1.68 avg-val = 26.80 avg-zval = 20.78 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 504000 bytes = 1449008 } seq expand size = 3528000 multi value expand size = 0 cnt WriteAppend = 1230 UnZipSize{ index = 0.0081 GB value = 0.0100 GB dict = 0.00 MB all = 0.0221 GB } __ZipSize{ index = 0.0008 GB value = 0.0105 GB dict = 0.00 MB all = 0.0128 GB } UnZip/Zip{ index = 9.5044 value = 0.9528 dict = 0.00 all = 1.7277 } Zip/UnZip{ index = 0.1052 value = 1.0496 dict = 0.00 all = 0.5788 } ---------------------------- total value len = 9.177883 GB avg = 0.257 KB (by entry num) total key len = 0.917899 GB avg = 0.026 KB total ukey len = 0.631681 GB avg = 0.018 KB total ukey num = 0.035777233 Billion total entry num = 0.035777233 Billion write speed all = 47.505797267 MB/sec (with seq num) write speed all = 46.158996312 MB/sec (without seq num) 2024/11/19-15:22:09.485780 139851179493120 [compaction_job.cc:1948] [default] [JOB 166] Generated table #4964: 504000 keys, 12776925 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:09.485829 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000929485806, "cf_name": "default", "job": 166, "event": "table_file_creation", "file_number": 4964, "file_size": 12776925, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 121866423, "largest_seqno": 130142237, "table_properties": {"data_size": 10474064, "index_size": 848448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12096000, "raw_average_key_size": 24, "raw_value_size": 9979200, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 504000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000929, "oldest_key_time": 0, "file_creation_time": 1732000929, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4964, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:09.489575 139851179493120 [compaction_job.cc:703] job-00166: subcompact[0], size: 0.030819 G, files: /storage/mytopling/.rocksdb [004962,004963,004964] 2024/11/19-15:22:09.490819 139851179493120 (Original Log Time 2024/11/19-15:22:09.049089) EVENT_LOG_v1 {"time_micros": 1732000929049072, "job": 166, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4817, 4826, 4831], "score": 6.31259, "input_data_size": 49030420, "oldest_snapshot_seqno": -1} 2024/11/19-15:22:09.490829 139851179493120 (Original Log Time 2024/11/19-15:22:09.490389) [compaction_job.cc:2022] [default] [JOB 166] Compacted 3@1 files to L2 => 30819416 bytes 2024/11/19-15:22:09.490833 139851179493120 (Original Log Time 2024/11/19-15:22:09.490700) [compaction_job.cc:1224] [default] compacted to: files[3 101 52 34 0 0 0] max score 4.86 (134 files need compaction), MB/sec: 111.3 rd, 70.0 wr, level 2, files in(3, 0) out(3 +0 blob) MB in(46.8, 0.0 +0.0 blob) out(29.4 +0.0 blob), read-write-amplify(1.6) write-amplify(0.6) OK, records in: 943292, records dropped: 0 output_compression: Snappy 2024/11/19-15:22:09.490859 139851179493120 (Original Log Time 2024/11/19-15:22:09.490723) EVENT_LOG_v1 {"time_micros": 1732000929490711, "cf": "default", "job": 166, "event": "compaction_finished", "compaction_time_micros": 440498, "compaction_time_cpu_micros": 430395, "output_level": 2, "num_output_files": 3, "total_output_size": 30819416, "num_input_records": 943292, "num_output_records": 943292, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 101, 52, 34, 0, 0, 0]} 2024/11/19-15:22:09.497500 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004600.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:09.497525 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000929497521, "job": 166, "event": "table_file_deletion", "file_number": 4600} 2024/11/19-15:22:09.501554 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004601.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:09.501576 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000929501574, "job": 166, "event": "table_file_deletion", "file_number": 4601} 2024/11/19-15:22:09.508331 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004770.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:09.508353 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000929508350, "job": 166, "event": "table_file_deletion", "file_number": 4770} 2024/11/19-15:22:09.515327 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004771.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:09.515349 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000929515346, "job": 166, "event": "table_file_deletion", "file_number": 4771} 2024/11/19-15:22:09.515517 139851179493120 [compaction_job.cc:2437] [default] [JOB 174] Compacting 4@1 files to L2, score 4.86, subcompactions 1 : 1 2024/11/19-15:22:09.515525 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 172 Base level 1, inputs: [4920(2051KB) 4818(31MB) 4821(31MB) 4823(31MB)] 2024/11/19-15:22:09.616665 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-00 2024/11/19-15:22:09.616743 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00172/att-00/compact.done) = No such file or directory: 7.213 sec on http probe failed: 3, retry 2024/11/19-15:22:09.630501 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-00 2024/11/19-15:22:09.630561 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00169/att-00/compact.done) = No such file or directory: 11.230 sec on http probe failed: 5, giveup 2024/11/19-15:22:09.630578 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00169/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.541 (submit 301.641 ms wait 11.230 s), input 127.820 MiB, send shutdown 2024/11/19-15:22:09.635645 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-00 2024/11/19-15:22:09.635859 139851158521600 [dcompact_etcd.cc:1773] job-00169/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:09.714373 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-01 2024/11/19-15:22:09.714447 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00168/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:22:09.794631 139851179493120 [dcompact_etcd.cc:1163] job-00174/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.279 sec, input = 97.103 MiB 2024/11/19-15:22:09.938205 139851158521600 [dcompact_etcd.cc:1163] job-00169/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.302 sec, input = 127.820 MiB 2024/11/19-15:22:10.390907 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-01 2024/11/19-15:22:10.391011 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00173/att-01/compact.done) = No such file or directory: 1.418 sec on http probe failed: 1, retry 2024/11/19-15:22:10.993147 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-01 2024/11/19-15:22:10.993240 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00167/att-01/compact.done) = No such file or directory: 5.206 sec on http probe failed: 2, retry 2024/11/19-15:22:11.179183 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-00 2024/11/19-15:22:11.179281 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00170/att-00/compact.done) = No such file or directory: 11.225 sec on http probe failed: 5, giveup 2024/11/19-15:22:11.179301 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00170/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.537 (submit 302.144 ms wait 11.225 s), input 127.820 MiB, send shutdown 2024/11/19-15:22:11.187387 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-00 2024/11/19-15:22:11.201482 139851148035840 [dcompact_etcd.cc:1773] job-00170/att-00: CleanFiles 0.014 sec : 0.000 2024/11/19-15:22:11.459704 139851148035840 [dcompact_etcd.cc:1163] job-00170/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.258 sec, input = 127.820 MiB 2024/11/19-15:22:11.530416 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-01 2024/11/19-15:22:11.530482 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00173/att-01/compact.done) = No such file or directory: 2.561 sec on http probe failed: 2, retry 2024/11/19-15:22:11.624817 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-00 2024/11/19-15:22:11.624914 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00172/att-00/compact.done) = No such file or directory: 9.222 sec on http probe failed: 4, retry 2024/11/19-15:22:11.720249 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-01 2024/11/19-15:22:11.720333 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00168/att-01/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:22:12.226998 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-00 2024/11/19-15:22:12.227073 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00174/att-00/compact.done) = No such file or directory: 2.428 sec on http probe failed: 1, retry 2024/11/19-15:22:12.672668 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-01 2024/11/19-15:22:12.672760 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00173/att-01/compact.done) = No such file or directory: 3.700 sec on http probe failed: 3, retry 2024/11/19-15:22:12.998535 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-01 2024/11/19-15:22:12.998630 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00167/att-01/compact.done) = No such file or directory: 7.214 sec on http probe failed: 3, retry 2024/11/19-15:22:13.142562 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-01 2024/11/19-15:22:13.142638 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00169/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:22:13.634431 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-00 2024/11/19-15:22:13.634544 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00172/att-00/compact.done) = No such file or directory: 11.230 sec on http probe failed: 5, giveup 2024/11/19-15:22:13.634569 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00172/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.821 sec, real 11.554 (submit 313.933 ms wait 11.230 s), input 127.821 MiB, send shutdown 2024/11/19-15:22:13.639282 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-00 2024/11/19-15:22:13.639535 139851137550080 [dcompact_etcd.cc:1773] job-00172/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:13.727968 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-01 2024/11/19-15:22:13.728028 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00168/att-01/compact.done) = No such file or directory: 7.211 sec on http probe failed: 3, retry 2024/11/19-15:22:13.815268 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-01 2024/11/19-15:22:13.815361 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00173/att-01/compact.done) = No such file or directory: 4.843 sec on http probe failed: 4, retry 2024/11/19-15:22:13.947024 139851137550080 [dcompact_etcd.cc:1163] job-00172/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.307 sec, input = 127.821 MiB 2024/11/19-15:22:14.174241 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-00 2024/11/19-15:22:14.174344 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00174/att-00/compact.done) = No such file or directory: 4.375 sec on http probe failed: 2, retry 2024/11/19-15:22:14.664660 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-01 2024/11/19-15:22:14.664759 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00170/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:22:14.957888 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-01 2024/11/19-15:22:14.957999 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00173/att-01/compact.done) = No such file or directory: 5.985 sec on http probe failed: 5, giveup 2024/11/19-15:22:14.958019 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00173/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 56.703 sec, real 6.248 (submit 254.588 ms wait 5.985 s), input 56.703 MiB, send shutdown 2024/11/19-15:22:14.962479 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-01 2024/11/19-15:22:14.962699 139850978166528 [dcompact_etcd.cc:1773] job-00173/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:15.006884 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-01 2024/11/19-15:22:15.006979 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00167/att-01/compact.done) = No such file or directory: 9.219 sec on http probe failed: 4, retry 2024/11/19-15:22:15.150406 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-01 2024/11/19-15:22:15.150511 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00169/att-01/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:22:15.311773 139850978166528 [dcompact_etcd.cc:1163] job-00173/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.349 sec, input = 56.703 MiB 2024/11/19-15:22:15.733518 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-01 2024/11/19-15:22:15.733594 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00168/att-01/compact.done) = No such file or directory: 9.219 sec on http probe failed: 4, retry 2024/11/19-15:22:16.124970 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-00 2024/11/19-15:22:16.125072 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00174/att-00/compact.done) = No such file or directory: 6.322 sec on http probe failed: 3, retry 2024/11/19-15:22:16.670035 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-01 2024/11/19-15:22:16.670129 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00170/att-01/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:22:16.738343 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-02 2024/11/19-15:22:16.738436 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00173/att-02/compact.done) = No such file or directory: 1.418 sec on http probe failed: 1, retry 2024/11/19-15:22:17.015527 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-01 2024/11/19-15:22:17.015607 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00167/att-01/compact.done) = No such file or directory: 11.228 sec on http probe failed: 5, giveup 2024/11/19-15:22:17.015628 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00167/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.541 (submit 304.358 ms wait 11.228 s), input 127.820 MiB, send shutdown 2024/11/19-15:22:17.023912 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-01 2024/11/19-15:22:17.024131 139851169007360 [dcompact_etcd.cc:1773] job-00167/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:17.151067 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-01 2024/11/19-15:22:17.151186 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00172/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:22:17.159642 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-01 2024/11/19-15:22:17.159726 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00169/att-01/compact.done) = No such file or directory: 7.212 sec on http probe failed: 3, retry 2024/11/19-15:22:17.289068 139851169007360 [dcompact_etcd.cc:1163] job-00167/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.265 sec, input = 127.820 MiB 2024/11/19-15:22:17.741637 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-01 2024/11/19-15:22:17.741741 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00168/att-01/compact.done) = No such file or directory: 11.224 sec on http probe failed: 5, giveup 2024/11/19-15:22:17.741761 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00168/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.822 sec, real 11.515 (submit 282.536 ms wait 11.224 s), input 127.822 MiB, send shutdown 2024/11/19-15:22:17.750995 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-01 2024/11/19-15:22:17.751221 139851127064320 [dcompact_etcd.cc:1773] job-00168/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:17.880652 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-02 2024/11/19-15:22:17.880737 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00173/att-02/compact.done) = No such file or directory: 2.561 sec on http probe failed: 2, retry 2024/11/19-15:22:18.031549 139851127064320 [dcompact_etcd.cc:1163] job-00168/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 127.822 MiB 2024/11/19-15:22:18.059137 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4965. Immutable memtables: 0. 2024/11/19-15:22:18.059670 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 175] Syncing log #4960 2024/11/19-15:22:18.059786 139850957195008 (Original Log Time 2024/11/19-15:22:18.059467) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:22:18.059798 139850957195008 [flush_job.cc:879] [default] [JOB 175] Flushing memtable with next log file: 4965 2024/11/19-15:22:18.059824 139850957195008 EVENT_LOG_v1 {"time_micros": 1732000938059817, "job": 175, "event": "flush_started", "num_memtables": 1, "num_entries": 1194593, "num_deletes": 0, "total_data_size": 480552059, "memory_usage": 546045320, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:22:18.059831 139850957195008 [flush_job.cc:910] [default] [JOB 175] Level-0 flush table #4966: started 2024/11/19-15:22:18.060489 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000021.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004966.sst in ConvertToSST, mem_size = 520.749 M, time = 643.338 us 2024/11/19-15:22:18.075366 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-00 2024/11/19-15:22:18.075473 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00174/att-00/compact.done) = No such file or directory: 8.273 sec on http probe failed: 4, retry 2024/11/19-15:22:18.591085 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004966.sst): fsize = 520.751 M, time(ms): open: 0.046, seek: 0.019, finish: 0.047, meta: 0.013, Flush: 0.016, sync: 530.421, close: 0.010, all: 530.572 2024/11/19-15:22:18.592527 139850957195008 [flush_job.cc:1083] [default] [JOB 175] Flush lasted 532740 microseconds, and 472448 cpu microseconds. 2024/11/19-15:22:18.620325 139850957195008 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:22:18.620397 139850957195008 (Original Log Time 2024/11/19-15:22:18.591138) [flush_job.cc:1029] [default] [JOB 175] Level-0 flush table #4966: 546046746 bytes OK (needs compaction) 2024/11/19-15:22:18.620406 139850957195008 (Original Log Time 2024/11/19-15:22:18.592548) [memtable_list.cc:560] [default] Level-0 commit table #4966 started 2024/11/19-15:22:18.620412 139850957195008 (Original Log Time 2024/11/19-15:22:18.620283) [memtable_list.cc:759] [default] Level-0 commit table #4966: memtable #1 done 2024/11/19-15:22:18.620417 139850957195008 (Original Log Time 2024/11/19-15:22:18.620317) EVENT_LOG_v1 {"time_micros": 1732000938620308, "job": 175, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [4, 101, 52, 34, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:22:18.620423 139850957195008 (Original Log Time 2024/11/19-15:22:18.620364) [db_impl_compaction_flush.cc:310] [default] Level summary: files[4 101 52 34 0 0 0] max score 4.67 (131 files need compaction) 2024/11/19-15:22:18.675821 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-01 2024/11/19-15:22:18.675897 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00170/att-01/compact.done) = No such file or directory: 7.211 sec on http probe failed: 3, retry 2024/11/19-15:22:19.023523 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-02 2024/11/19-15:22:19.023645 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00173/att-02/compact.done) = No such file or directory: 3.703 sec on http probe failed: 3, retry 2024/11/19-15:22:19.160400 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-01 2024/11/19-15:22:19.160500 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00172/att-01/compact.done) = No such file or directory: 5.204 sec on http probe failed: 2, retry 2024/11/19-15:22:19.164599 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-01 2024/11/19-15:22:19.164702 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00169/att-01/compact.done) = No such file or directory: 9.222 sec on http probe failed: 4, retry 2024/11/19-15:22:20.025713 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-00 2024/11/19-15:22:20.025814 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00174/att-00/compact.done) = No such file or directory: 10.223 sec on http probe failed: 5, giveup 2024/11/19-15:22:20.025832 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00174/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 97.103 sec, real 10.510 (submit 278.664 ms wait 10.223 s), input 97.103 MiB, send shutdown 2024/11/19-15:22:20.033669 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-00 2024/11/19-15:22:20.033896 139851179493120 [dcompact_etcd.cc:1773] job-00174/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:20.165845 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-02 2024/11/19-15:22:20.165947 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00173/att-02/compact.done) = No such file or directory: 4.846 sec on http probe failed: 4, retry 2024/11/19-15:22:20.333252 139851179493120 [dcompact_etcd.cc:1163] job-00174/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.299 sec, input = 97.103 MiB 2024/11/19-15:22:20.493970 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-02 2024/11/19-15:22:20.494077 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00167/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:22:20.683250 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-01 2024/11/19-15:22:20.683324 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00170/att-01/compact.done) = No such file or directory: 9.216 sec on http probe failed: 4, retry 2024/11/19-15:22:21.166795 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-01 2024/11/19-15:22:21.166874 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00172/att-01/compact.done) = No such file or directory: 7.214 sec on http probe failed: 3, retry 2024/11/19-15:22:21.173261 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-01 2024/11/19-15:22:21.173344 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00169/att-01/compact.done) = No such file or directory: 11.227 sec on http probe failed: 5, giveup 2024/11/19-15:22:21.173369 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00169/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.537 (submit 302.253 ms wait 11.227 s), input 127.820 MiB, send shutdown 2024/11/19-15:22:21.178574 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-01 2024/11/19-15:22:21.178834 139851158521600 [dcompact_etcd.cc:1773] job-00169/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:21.236577 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-02 2024/11/19-15:22:21.236665 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00168/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:22:21.308409 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-02 2024/11/19-15:22:21.308488 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00173/att-02/compact.done) = No such file or directory: 5.988 sec on http probe failed: 5, giveup 2024/11/19-15:22:21.308507 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00173/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 56.703 sec, real 6.346 (submit 348.987 ms wait 5.988 s), input 56.703 MiB, send shutdown 2024/11/19-15:22:21.313352 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00173/att-02 2024/11/19-15:22:21.313556 139850978166528 [dcompact_etcd.cc:1773] job-00173/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:21.315501 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00173) = success 2024/11/19-15:22:21.315509 139850978166528 [dcompact_etcd.cc:1773] job-00173/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:22:21.315531 139850978166528 (Original Log Time 2024/11/19-15:22:02.398389) EVENT_LOG_v1 {"time_micros": 1732000922398382, "job": 173, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4912, 4916], "score": 4.97302, "input_data_size": 59163526, "oldest_snapshot_seqno": 134613078} 2024/11/19-15:22:21.315538 139850978166528 [compaction_job.cc:2437] [default] [JOB 173] Compacting 2@1 files to L2, score 4.97, subcompactions 1 : 1 2024/11/19-15:22:21.315545 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 171 Base level 1, inputs: [4912(31MB) 4916(24MB)] 2024/11/19-15:22:21.315779 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004967.sst) /storage/mytopling/..//tmp/Topling-82038-82DKdF 2024/11/19-15:22:21.445286 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004967.sst: sumWaitingMem = 0.005 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0053 GB, waited 0.000 sec, Key+Value = 0.059 GB 2024/11/19-15:22:21.471169 139851158521600 [dcompact_etcd.cc:1163] job-00169/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.292 sec, input = 127.820 MiB 2024/11/19-15:22:21.475215 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004967.sst): index pass time = 0.03's, 219.799'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 410699 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0066 GB zip-key = 0.0008 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:22:21.534151 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004967.sst): first pass time = 0.22's, 272.294'MB/sec 2024/11/19-15:22:21.534209 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004967.sst): old prealloc_size = 65079878, real_size = 49280960 2024/11/19-15:22:21.534420 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004967.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 410699 prefix = 4 raw-key = 0.0066 GB zip-key = 0.0008 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0480 GB zip-val = 0.0484 GB avg-val = 116.77 avg-zval = 117.85 2024/11/19-15:22:21.534482 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004967.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0010 GB, waited 0.089 sec, Key+Value = 0.059 GB 2024/11/19-15:22:21.617209 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004967.sst): second pass time = 0.00's,148019413.580'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.08's, 592.622'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3736.200'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.660'MB/sec write SST data time = 0.04's, 1113.101'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.048'GB zip my value throughput = 148019413.580'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 410699 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 117.85 usrkeys = 410699 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 5862 UnZipSize{ index = 0.0082 GB value = 0.0480 GB dict = 0.00 MB all = 0.0595 GB } __ZipSize{ index = 0.0008 GB value = 0.0484 GB dict = 0.00 MB all = 0.0492 GB } UnZip/Zip{ index = 10.5693 value = 0.9909 dict = 0.00 all = 1.2089 } Zip/UnZip{ index = 0.0946 value = 1.0092 dict = 0.00 all = 0.8272 } ---------------------------- total value len = 9.225841 GB avg = 0.255 KB (by entry num) total key len = 0.929398 GB avg = 0.026 KB total ukey len = 0.639895 GB avg = 0.018 KB total ukey num = 0.036187932 Billion total entry num = 0.036187932 Billion write speed all = 45.204907482 MB/sec (with seq num) write speed all = 43.916215329 MB/sec (without seq num) 2024/11/19-15:22:21.620323 139850978166528 [compaction_job.cc:1948] [default] [JOB 173] Generated table #4967: 410699 keys, 49183064 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:21.620373 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000941620351, "cf_name": "default", "job": 173, "event": "table_file_creation", "file_number": 4967, "file_size": 49183064, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 48400944, "index_size": 777152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 11499572, "raw_average_key_size": 28, "raw_value_size": 47958290, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 410699, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000941, "oldest_key_time": 0, "file_creation_time": 1732000941, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4967, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:21.632093 139850978166528 [compaction_job.cc:703] job-00173: subcompact[0], size: 0.049183 G, files: /storage/mytopling/.rocksdb [004967] 2024/11/19-15:22:21.632980 139850978166528 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:22:21.633099 139850978166528 (Original Log Time 2024/11/19-15:22:21.315573) EVENT_LOG_v1 {"time_micros": 1732000941315557, "job": 173, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4912, 4916], "score": 4.97302, "input_data_size": 59163526, "oldest_snapshot_seqno": 134613078} 2024/11/19-15:22:21.633105 139850978166528 (Original Log Time 2024/11/19-15:22:21.632561) [compaction_job.cc:2022] [default] [JOB 173] Compacted 2@1 files to L2 => 49183064 bytes 2024/11/19-15:22:21.633108 139850978166528 (Original Log Time 2024/11/19-15:22:21.632894) [compaction_job.cc:1224] [default] compacted to: files[4 99 53 34 0 0 0] max score 4.67 (132 files need compaction), MB/sec: 186.9 rd, 155.4 wr, level 2, files in(2, 0) out(1 +0 blob) MB in(56.4, 0.0 +0.0 blob) out(46.9 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 410699, records dropped: 0 output_compression: Snappy 2024/11/19-15:22:21.633127 139850978166528 (Original Log Time 2024/11/19-15:22:21.632973) EVENT_LOG_v1 {"time_micros": 1732000941632907, "cf": "default", "job": 173, "event": "compaction_finished", "compaction_time_micros": 316530, "compaction_time_cpu_micros": 315182, "output_level": 2, "num_output_files": 1, "total_output_size": 49183064, "num_input_records": 410699, "num_output_records": 410699, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 99, 53, 34, 0, 0, 0]} 2024/11/19-15:22:21.633209 139850978166528 [compaction_job.cc:2437] [default] [JOB 176] Compacting 4@1 files to L2, score 4.67, subcompactions 1 : 1 2024/11/19-15:22:21.633218 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 174 Base level 1, inputs: [4825(31MB) 4828(31MB) 4830(31MB) 4832(31MB)] 2024/11/19-15:22:21.949110 139850978166528 [dcompact_etcd.cc:1163] job-00176/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.315 sec, input = 126.936 MiB 2024/11/19-15:22:22.503748 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-02 2024/11/19-15:22:22.503834 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00167/att-02/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:22:22.688753 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-01 2024/11/19-15:22:22.688837 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00170/att-01/compact.done) = No such file or directory: 11.224 sec on http probe failed: 5, giveup 2024/11/19-15:22:22.688856 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00170/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.487 (submit 258.119 ms wait 11.224 s), input 127.820 MiB, send shutdown 2024/11/19-15:22:22.698081 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-01 2024/11/19-15:22:22.698307 139851148035840 [dcompact_etcd.cc:1773] job-00170/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:22.770096 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-01 2024/11/19-15:22:22.770172 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00174/att-01/compact.done) = No such file or directory: 2.428 sec on http probe failed: 1, retry 2024/11/19-15:22:22.973838 139851148035840 [dcompact_etcd.cc:1163] job-00170/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.275 sec, input = 127.820 MiB 2024/11/19-15:22:23.174109 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-01 2024/11/19-15:22:23.174184 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00172/att-01/compact.done) = No such file or directory: 9.220 sec on http probe failed: 4, retry 2024/11/19-15:22:23.245012 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-02 2024/11/19-15:22:23.245074 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00168/att-02/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:22:24.512872 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-02 2024/11/19-15:22:24.512982 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00167/att-02/compact.done) = No such file or directory: 7.215 sec on http probe failed: 3, retry 2024/11/19-15:22:24.673108 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-02 2024/11/19-15:22:24.673200 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00169/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:22:24.719852 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-01 2024/11/19-15:22:24.719955 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00174/att-01/compact.done) = No such file or directory: 4.379 sec on http probe failed: 2, retry 2024/11/19-15:22:25.132073 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-00 2024/11/19-15:22:25.132161 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00176/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:22:25.180879 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-01 2024/11/19-15:22:25.180951 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00172/att-01/compact.done) = No such file or directory: 11.227 sec on http probe failed: 5, giveup 2024/11/19-15:22:25.180969 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00172/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.821 sec, real 11.541 (submit 307.341 ms wait 11.227 s), input 127.821 MiB, send shutdown 2024/11/19-15:22:25.186213 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-01 2024/11/19-15:22:25.186426 139851137550080 [dcompact_etcd.cc:1773] job-00172/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:25.249680 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-02 2024/11/19-15:22:25.249743 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00168/att-02/compact.done) = No such file or directory: 7.214 sec on http probe failed: 3, retry 2024/11/19-15:22:25.483943 139851137550080 [dcompact_etcd.cc:1163] job-00172/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.297 sec, input = 127.821 MiB 2024/11/19-15:22:26.178786 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-02 2024/11/19-15:22:26.178888 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00170/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:22:26.522165 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-02 2024/11/19-15:22:26.522265 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00167/att-02/compact.done) = No such file or directory: 9.224 sec on http probe failed: 4, retry 2024/11/19-15:22:26.667157 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-01 2024/11/19-15:22:26.667238 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00174/att-01/compact.done) = No such file or directory: 6.329 sec on http probe failed: 3, retry 2024/11/19-15:22:26.681978 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-02 2024/11/19-15:22:26.682041 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00169/att-02/compact.done) = No such file or directory: 5.202 sec on http probe failed: 2, retry 2024/11/19-15:22:27.141474 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-00 2024/11/19-15:22:27.141568 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00176/att-00/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:22:27.255551 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-02 2024/11/19-15:22:27.255627 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00168/att-02/compact.done) = No such file or directory: 9.218 sec on http probe failed: 4, retry 2024/11/19-15:22:28.014281 139852349699840 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:22:28.186654 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-02 2024/11/19-15:22:28.186748 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00170/att-02/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:22:28.530957 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-02 2024/11/19-15:22:28.531042 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00167/att-02/compact.done) = No such file or directory: 11.233 sec on http probe failed: 5, giveup 2024/11/19-15:22:28.531061 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00167/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.507 (submit 264.863 ms wait 11.233 s), input 127.820 MiB, send shutdown 2024/11/19-15:22:28.539001 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00167/att-02 2024/11/19-15:22:28.539231 139851169007360 [dcompact_etcd.cc:1773] job-00167/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:28.541183 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00167) = success 2024/11/19-15:22:28.541202 139851169007360 [dcompact_etcd.cc:1773] job-00167/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:22:28.541231 139851169007360 (Original Log Time 2024/11/19-15:21:53.919365) EVENT_LOG_v1 {"time_micros": 1732000913919354, "job": 167, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4841, 4846, 4849, 4853], "score": 6.22126, "input_data_size": 133300172, "oldest_snapshot_seqno": -1} 2024/11/19-15:22:28.541242 139851169007360 [compaction_job.cc:2437] [default] [JOB 167] Compacting 4@1 files to L2, score 6.22, subcompactions 1 : 1 2024/11/19-15:22:28.541254 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 165 Base level 1, inputs: [4841(31MB) 4846(31MB) 4849(31MB) 4853(31MB)] 2024/11/19-15:22:28.541555 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004968.sst) /storage/mytopling/..//tmp/Topling-82038-AGNvkg 2024/11/19-15:22:28.614721 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-01 2024/11/19-15:22:28.614796 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00174/att-01/compact.done) = No such file or directory: 8.276 sec on http probe failed: 4, retry 2024/11/19-15:22:28.684956 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-02 2024/11/19-15:22:28.685035 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00172/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:22:28.690777 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-02 2024/11/19-15:22:28.690834 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00169/att-02/compact.done) = No such file or directory: 7.211 sec on http probe failed: 3, retry 2024/11/19-15:22:28.704900 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004968.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:22:28.740013 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004968.sst): index pass time = 0.04's, 211.316'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463355 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:22:28.804911 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004968.sst): first pass time = 0.26's, 254.841'MB/sec 2024/11/19-15:22:28.804972 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004968.sst): old prealloc_size = 73819750, real_size = 55605904 2024/11/19-15:22:28.805202 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004968.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463355 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.83 avg-zval = 117.87 2024/11/19-15:22:28.805258 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004968.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.100 sec, Key+Value = 0.067 GB 2024/11/19-15:22:28.898056 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004968.sst): second pass time = 0.00's,153792710.227'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 596.288'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3861.996'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.646'MB/sec write SST data time = 0.05's, 1131.000'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 153792710.227'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463355 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.83 avg-zval = 117.87 usrkeys = 463355 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.83 avg-zval = 117.87 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6615 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5731 value = 0.9912 dict = 0.00 all = 1.2093 } Zip/UnZip{ index = 0.0946 value = 1.0088 dict = 0.00 all = 0.8269 } ---------------------------- total value len = 9.279976 GB avg = 0.253 KB (by entry num) total key len = 0.942372 GB avg = 0.026 KB total ukey len = 0.649162 GB avg = 0.018 KB total ukey num = 0.036651287 Billion total entry num = 0.036651287 Billion write speed all = 44.075165702 MB/sec (with seq num) write speed all = 42.810946143 MB/sec (without seq num) 2024/11/19-15:22:28.901449 139851169007360 [compaction_job.cc:1948] [default] [JOB 167] Generated table #4968: 463355 keys, 55495320 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:28.901510 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000948901485, "cf_name": "default", "job": 167, "event": "table_file_creation", "file_number": 4968, "file_size": 55495320, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54613376, "index_size": 876480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12973940, "raw_average_key_size": 28, "raw_value_size": 54135034, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463355, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000948, "oldest_key_time": 0, "file_creation_time": 1732000948, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4968, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:28.914849 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004969.sst) /storage/mytopling/..//tmp/Topling-82038-LhAP6i 2024/11/19-15:22:29.061333 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004969.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:22:29.095592 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004969.sst): index pass time = 0.03's, 216.085'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462181 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:22:29.150066 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-00 2024/11/19-15:22:29.150162 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00176/att-00/compact.done) = No such file or directory: 7.193 sec on http probe failed: 3, retry 2024/11/19-15:22:29.162603 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004969.sst): first pass time = 0.25's, 270.123'MB/sec 2024/11/19-15:22:29.162651 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004969.sst): old prealloc_size = 73819750, real_size = 55466256 2024/11/19-15:22:29.162876 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004969.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462181 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.79 avg-zval = 117.85 2024/11/19-15:22:29.162949 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004969.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.102 sec, Key+Value = 0.067 GB 2024/11/19-15:22:29.255832 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004969.sst): second pass time = 0.00's,131977550.122'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 594.251'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3979.997'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.628'MB/sec write SST data time = 0.05's, 1130.552'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 131977550.122'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462181 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 117.85 usrkeys = 462181 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6596 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0554 GB } UnZip/Zip{ index = 10.5011 value = 0.9910 dict = 0.00 all = 1.2089 } Zip/UnZip{ index = 0.0952 value = 1.0091 dict = 0.00 all = 0.8272 } ---------------------------- total value len = 9.333955 GB avg = 0.251 KB (by entry num) total key len = 0.955313 GB avg = 0.026 KB total ukey len = 0.658406 GB avg = 0.018 KB total ukey num = 0.037113468 Billion total entry num = 0.037113468 Billion write speed all = 44.295372893 MB/sec (with seq num) write speed all = 43.017182911 MB/sec (without seq num) 2024/11/19-15:22:29.259263 139851169007360 [compaction_job.cc:1948] [default] [JOB 167] Generated table #4969: 462181 keys, 55355992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:29.259316 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000949259292, "cf_name": "default", "job": 167, "event": "table_file_creation", "file_number": 4969, "file_size": 55355992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54470272, "index_size": 880256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12941068, "raw_average_key_size": 28, "raw_value_size": 53978818, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462181, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000948, "oldest_key_time": 0, "file_creation_time": 1732000948, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4969, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:29.263970 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-02 2024/11/19-15:22:29.264052 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00168/att-02/compact.done) = No such file or directory: 11.224 sec on http probe failed: 5, giveup 2024/11/19-15:22:29.264069 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00168/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.822 sec, real 11.513 (submit 280.256 ms wait 11.224 s), input 127.822 MiB, send shutdown 2024/11/19-15:22:29.271969 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00168/att-02 2024/11/19-15:22:29.272160 139851127064320 [dcompact_etcd.cc:1773] job-00168/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:29.272739 139851169007360 [compaction_job.cc:703] job-00167: subcompact[0], size: 0.110851 G, files: /storage/mytopling/.rocksdb [004968,004969] 2024/11/19-15:22:29.274023 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00168) = success 2024/11/19-15:22:29.274031 139851127064320 [dcompact_etcd.cc:1773] job-00168/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:22:29.274043 139851169007360 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:22:29.274049 139851127064320 (Original Log Time 2024/11/19-15:21:54.679360) EVENT_LOG_v1 {"time_micros": 1732000914679343, "job": 168, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4857, 4861, 4865, 4869], "score": 5.97162, "input_data_size": 133300172, "oldest_snapshot_seqno": 133904784} 2024/11/19-15:22:29.274057 139851127064320 [compaction_job.cc:2437] [default] [JOB 168] Compacting 4@1 files to L2, score 5.97, subcompactions 1 : 1 2024/11/19-15:22:29.274064 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 166 Base level 1, inputs: [4857(31MB) 4861(31MB) 4865(31MB) 4869(31MB)] 2024/11/19-15:22:29.274160 139851169007360 (Original Log Time 2024/11/19-15:22:28.541296) EVENT_LOG_v1 {"time_micros": 1732000948541271, "job": 167, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4841, 4846, 4849, 4853], "score": 6.22126, "input_data_size": 133300172, "oldest_snapshot_seqno": -1} 2024/11/19-15:22:29.274167 139851169007360 (Original Log Time 2024/11/19-15:22:29.273666) [compaction_job.cc:2022] [default] [JOB 167] Compacted 4@1 files to L2 => 110851312 bytes 2024/11/19-15:22:29.274170 139851169007360 (Original Log Time 2024/11/19-15:22:29.274013) [compaction_job.cc:1224] [default] compacted to: files[4 95 55 34 0 0 0] max score 4.42 (130 files need compaction), MB/sec: 182.2 rd, 151.5 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925536, records dropped: 0 output_compression: Snappy 2024/11/19-15:22:29.274188 139851169007360 (Original Log Time 2024/11/19-15:22:29.274037) EVENT_LOG_v1 {"time_micros": 1732000949274025, "cf": "default", "job": 167, "event": "compaction_finished", "compaction_time_micros": 731454, "compaction_time_cpu_micros": 728593, "output_level": 2, "num_output_files": 2, "total_output_size": 110851312, "num_input_records": 925536, "num_output_records": 925536, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 95, 55, 34, 0, 0, 0]} 2024/11/19-15:22:29.274260 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004970.sst) /storage/mytopling/..//tmp/Topling-82038-MgoKa9 2024/11/19-15:22:29.275145 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004602.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:29.275165 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000949275162, "job": 167, "event": "table_file_deletion", "file_number": 4602} 2024/11/19-15:22:29.278909 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004603.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:29.278940 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000949278938, "job": 167, "event": "table_file_deletion", "file_number": 4603} 2024/11/19-15:22:29.286250 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004606.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:29.286270 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000949286268, "job": 167, "event": "table_file_deletion", "file_number": 4606} 2024/11/19-15:22:29.291418 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004769.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:29.291437 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000949291435, "job": 167, "event": "table_file_deletion", "file_number": 4769} 2024/11/19-15:22:29.291596 139851169007360 [compaction_job.cc:2437] [default] [JOB 177] Compacting 4@1 files to L2, score 4.42, subcompactions 1 : 1 2024/11/19-15:22:29.291605 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 175 Base level 1, inputs: [4835(31MB) 4837(31MB) 4839(31MB) 4842(31MB)] 2024/11/19-15:22:29.433831 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004970.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:22:29.485099 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004970.sst): index pass time = 0.05's, 144.751'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463444 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:22:29.535332 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004970.sst): first pass time = 0.26's, 257.063'MB/sec 2024/11/19-15:22:29.535387 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004970.sst): old prealloc_size = 73819750, real_size = 55613504 2024/11/19-15:22:29.535626 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004970.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463444 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.80 avg-zval = 117.86 2024/11/19-15:22:29.535688 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004970.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.102 sec, Key+Value = 0.067 GB 2024/11/19-15:22:29.601657 139851169007360 [dcompact_etcd.cc:1163] job-00177/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.310 sec, input = 126.970 MiB 2024/11/19-15:22:29.629875 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004970.sst): second pass time = 0.00's,109358537.374'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 587.476'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3727.606'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.592'MB/sec write SST data time = 0.05's, 1120.289'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 109358537.374'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463444 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.80 avg-zval = 117.86 usrkeys = 463444 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.80 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6615 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5713 value = 0.9911 dict = 0.00 all = 1.2091 } Zip/UnZip{ index = 0.0946 value = 1.0090 dict = 0.00 all = 0.8271 } ---------------------------- total value len = 9.388087 GB avg = 0.250 KB (by entry num) total key len = 0.968290 GB avg = 0.026 KB total ukey len = 0.667674 GB avg = 0.018 KB total ukey num = 0.037576912 Billion total entry num = 0.037576912 Billion write speed all = 44.512597279 MB/sec (with seq num) write speed all = 43.220526918 MB/sec (without seq num) 2024/11/19-15:22:29.633230 139851127064320 [compaction_job.cc:1948] [default] [JOB 168] Generated table #4970: 463444 keys, 55502936 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:29.633279 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000949633256, "cf_name": "default", "job": 168, "event": "table_file_creation", "file_number": 4970, "file_size": 55502936, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54620656, "index_size": 876800, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12976432, "raw_average_key_size": 28, "raw_value_size": 54132476, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463444, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000949, "oldest_key_time": 0, "file_creation_time": 1732000949, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4970, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:29.646875 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004971.sst) /storage/mytopling/..//tmp/Topling-82038-TlYnD4 2024/11/19-15:22:29.791925 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004971.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:22:29.825873 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004971.sst): index pass time = 0.03's, 218.037'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462109 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:22:29.895365 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004971.sst): first pass time = 0.25's, 269.332'MB/sec 2024/11/19-15:22:29.895421 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004971.sst): old prealloc_size = 73819750, real_size = 55460704 2024/11/19-15:22:29.895653 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004971.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462109 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.82 avg-zval = 117.86 2024/11/19-15:22:29.895715 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004971.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.104 sec, Key+Value = 0.067 GB 2024/11/19-15:22:29.990437 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004971.sst): second pass time = 0.00's,124960337.963'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 582.628'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3825.118'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.584'MB/sec write SST data time = 0.05's, 1105.771'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 124960337.963'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462109 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.82 avg-zval = 117.86 usrkeys = 462109 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.82 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6597 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0554 GB } UnZip/Zip{ index = 10.5017 value = 0.9911 dict = 0.00 all = 1.2091 } Zip/UnZip{ index = 0.0952 value = 1.0089 dict = 0.00 all = 0.8271 } ---------------------------- total value len = 9.442070 GB avg = 0.248 KB (by entry num) total key len = 0.981229 GB avg = 0.026 KB total ukey len = 0.676917 GB avg = 0.018 KB total ukey num = 0.038039021 Billion total entry num = 0.038039021 Billion write speed all = 44.730913363 MB/sec (with seq num) write speed all = 43.424977354 MB/sec (without seq num) 2024/11/19-15:22:29.993968 139851127064320 [compaction_job.cc:1948] [default] [JOB 168] Generated table #4971: 462109 keys, 55350488 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:29.994019 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000949993996, "cf_name": "default", "job": 168, "event": "table_file_creation", "file_number": 4971, "file_size": 55350488, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54464912, "index_size": 880064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12939052, "raw_average_key_size": 28, "raw_value_size": 53982866, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462109, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000949, "oldest_key_time": 0, "file_creation_time": 1732000949, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4971, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:30.007855 139851127064320 [compaction_job.cc:703] job-00168: subcompact[0], size: 0.110853 G, files: /storage/mytopling/.rocksdb [004970,004971] 2024/11/19-15:22:30.009184 139851127064320 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:22:30.009290 139851127064320 (Original Log Time 2024/11/19-15:22:29.274089) EVENT_LOG_v1 {"time_micros": 1732000949274074, "job": 168, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4857, 4861, 4865, 4869], "score": 5.97162, "input_data_size": 133300172, "oldest_snapshot_seqno": 133904784} 2024/11/19-15:22:30.009296 139851127064320 (Original Log Time 2024/11/19-15:22:30.008814) [compaction_job.cc:2022] [default] [JOB 168] Compacted 4@1 files to L2 => 110853424 bytes 2024/11/19-15:22:30.009299 139851127064320 (Original Log Time 2024/11/19-15:22:30.009150) [compaction_job.cc:1224] [default] compacted to: files[4 91 57 34 0 0 0] max score 4.17 (128 files need compaction), MB/sec: 181.7 rd, 151.1 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925553, records dropped: 0 output_compression: Snappy 2024/11/19-15:22:30.009319 139851127064320 (Original Log Time 2024/11/19-15:22:30.009177) EVENT_LOG_v1 {"time_micros": 1732000950009163, "cf": "default", "job": 168, "event": "compaction_finished", "compaction_time_micros": 733781, "compaction_time_cpu_micros": 727873, "output_level": 2, "num_output_files": 2, "total_output_size": 110853424, "num_input_records": 925553, "num_output_records": 925553, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 91, 57, 34, 0, 0, 0]} 2024/11/19-15:22:30.016132 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004779.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:30.016160 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000950016157, "job": 168, "event": "table_file_deletion", "file_number": 4779} 2024/11/19-15:22:30.023133 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004780.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:30.023157 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000950023155, "job": 168, "event": "table_file_deletion", "file_number": 4780} 2024/11/19-15:22:30.029854 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004781.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:30.029877 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000950029875, "job": 168, "event": "table_file_deletion", "file_number": 4781} 2024/11/19-15:22:30.036403 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004782.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:30.036430 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000950036428, "job": 168, "event": "table_file_deletion", "file_number": 4782} 2024/11/19-15:22:30.036600 139851127064320 [compaction_job.cc:2437] [default] [JOB 178] Compacting 4@1 files to L2, score 4.17, subcompactions 1 : 1 2024/11/19-15:22:30.036610 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 176 Base level 1, inputs: [4845(31MB) 4847(31MB) 4848(31MB) 4850(31MB)] 2024/11/19-15:22:30.192192 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-02 2024/11/19-15:22:30.192260 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00170/att-02/compact.done) = No such file or directory: 7.213 sec on http probe failed: 3, retry 2024/11/19-15:22:30.313749 139851127064320 [dcompact_etcd.cc:1163] job-00178/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.277 sec, input = 126.907 MiB 2024/11/19-15:22:30.562521 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-01 2024/11/19-15:22:30.562609 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00174/att-01/compact.done) = No such file or directory: 10.224 sec on http probe failed: 5, giveup 2024/11/19-15:22:30.562627 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00174/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 97.103 sec, real 10.529 (submit 299.266 ms wait 10.224 s), input 97.103 MiB, send shutdown 2024/11/19-15:22:30.570501 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-01 2024/11/19-15:22:30.570718 139851179493120 [dcompact_etcd.cc:1773] job-00174/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:30.693681 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-02 2024/11/19-15:22:30.693758 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00172/att-02/compact.done) = No such file or directory: 5.201 sec on http probe failed: 2, retry 2024/11/19-15:22:30.698754 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-02 2024/11/19-15:22:30.698809 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00169/att-02/compact.done) = No such file or directory: 9.220 sec on http probe failed: 4, retry 2024/11/19-15:22:30.864942 139851179493120 [dcompact_etcd.cc:1163] job-00174/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.294 sec, input = 97.103 MiB 2024/11/19-15:22:31.155616 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-00 2024/11/19-15:22:31.155693 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00176/att-00/compact.done) = No such file or directory: 9.201 sec on http probe failed: 4, retry 2024/11/19-15:22:32.202451 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-02 2024/11/19-15:22:32.202553 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00170/att-02/compact.done) = No such file or directory: 9.219 sec on http probe failed: 4, retry 2024/11/19-15:22:32.698602 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-02 2024/11/19-15:22:32.698700 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00172/att-02/compact.done) = No such file or directory: 7.210 sec on http probe failed: 3, retry 2024/11/19-15:22:32.707253 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-02 2024/11/19-15:22:32.707314 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00169/att-02/compact.done) = No such file or directory: 11.228 sec on http probe failed: 5, giveup 2024/11/19-15:22:32.707332 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00169/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.528 (submit 292.211 ms wait 11.228 s), input 127.820 MiB, send shutdown 2024/11/19-15:22:32.712987 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00169/att-02 2024/11/19-15:22:32.713208 139851158521600 [dcompact_etcd.cc:1773] job-00169/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:32.715092 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00169) = success 2024/11/19-15:22:32.715101 139851158521600 [dcompact_etcd.cc:1773] job-00169/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:22:32.715121 139851158521600 (Original Log Time 2024/11/19-15:21:58.089188) EVENT_LOG_v1 {"time_micros": 1732000918089180, "job": 169, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4872, 4874, 4876, 4880], "score": 5.72196, "input_data_size": 133300108, "oldest_snapshot_seqno": -1} 2024/11/19-15:22:32.715128 139851158521600 [compaction_job.cc:2437] [default] [JOB 169] Compacting 4@1 files to L2, score 5.72, subcompactions 1 : 1 2024/11/19-15:22:32.715136 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 167 Base level 1, inputs: [4872(31MB) 4874(31MB) 4876(31MB) 4880(31MB)] 2024/11/19-15:22:32.715360 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004972.sst) /storage/mytopling/..//tmp/Topling-82038-KfonR1 2024/11/19-15:22:32.784238 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-00 2024/11/19-15:22:32.784335 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00177/att-00/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:22:32.857487 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004972.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:22:32.891691 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004972.sst): index pass time = 0.03's, 217.011'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463550 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:22:32.970418 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004972.sst): first pass time = 0.26's, 263.127'MB/sec 2024/11/19-15:22:32.970473 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004972.sst): old prealloc_size = 73819750, real_size = 55622304 2024/11/19-15:22:32.970709 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004972.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463550 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.77 avg-zval = 117.85 2024/11/19-15:22:32.970777 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004972.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.113 sec, Key+Value = 0.067 GB 2024/11/19-15:22:33.064640 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004972.sst): second pass time = 0.00's,106344950.884'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 589.618'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3789.237'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.630'MB/sec write SST data time = 0.05's, 1113.505'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 106344950.884'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463550 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 117.85 usrkeys = 463550 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6615 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5721 value = 0.9909 dict = 0.00 all = 1.2089 } Zip/UnZip{ index = 0.0946 value = 1.0092 dict = 0.00 all = 0.8272 } ---------------------------- total value len = 9.496200 GB avg = 0.247 KB (by entry num) total key len = 0.994208 GB avg = 0.026 KB total ukey len = 0.686188 GB avg = 0.018 KB total ukey num = 0.038502571 Billion total entry num = 0.038502571 Billion write speed all = 44.432718482 MB/sec (with seq num) write speed all = 43.128079855 MB/sec (without seq num) 2024/11/19-15:22:33.086079 139851158521600 [compaction_job.cc:1948] [default] [JOB 169] Generated table #4972: 463550 keys, 55511768 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:33.086136 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000953086113, "cf_name": "default", "job": 169, "event": "table_file_creation", "file_number": 4972, "file_size": 55511768, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54629328, "index_size": 876928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12979400, "raw_average_key_size": 28, "raw_value_size": 54129580, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463550, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000952, "oldest_key_time": 0, "file_creation_time": 1732000952, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4972, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:33.100128 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004973.sst) /storage/mytopling/..//tmp/Topling-82038-dz1m0E 2024/11/19-15:22:33.160914 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-00 2024/11/19-15:22:33.161017 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00176/att-00/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:22:33.161035 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00176/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.936 sec, real 11.527 (submit 315.446 ms wait 11.207 s), input 126.936 MiB, send shutdown 2024/11/19-15:22:33.166762 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-00 2024/11/19-15:22:33.196885 139850978166528 [dcompact_etcd.cc:1773] job-00176/att-00: CleanFiles 0.030 sec : 0.000 2024/11/19-15:22:33.241860 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004973.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:22:33.291136 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004973.sst): index pass time = 0.05's, 150.143'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462086 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:22:33.301975 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-02 2024/11/19-15:22:33.302077 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00174/att-02/compact.done) = No such file or directory: 2.428 sec on http probe failed: 1, retry 2024/11/19-15:22:33.341997 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004973.sst): first pass time = 0.24's, 276.698'MB/sec 2024/11/19-15:22:33.342048 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004973.sst): old prealloc_size = 73819750, real_size = 55452304 2024/11/19-15:22:33.342275 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004973.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462086 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.82 avg-zval = 117.86 2024/11/19-15:22:33.342321 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004973.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.100 sec, Key+Value = 0.067 GB 2024/11/19-15:22:33.434501 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004973.sst): second pass time = 0.00's,174133200.000'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 598.695'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3905.162'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.635'MB/sec write SST data time = 0.05's, 1144.030'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 174133200.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462086 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.82 avg-zval = 117.86 usrkeys = 462086 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.82 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6596 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0553 GB } UnZip/Zip{ index = 10.5727 value = 0.9912 dict = 0.00 all = 1.2092 } Zip/UnZip{ index = 0.0946 value = 1.0089 dict = 0.00 all = 0.8270 } ---------------------------- total value len = 9.550181 GB avg = 0.245 KB (by entry num) total key len = 1.007147 GB avg = 0.026 KB total ukey len = 0.695429 GB avg = 0.018 KB total ukey num = 0.038964657 Billion total entry num = 0.038964657 Billion write speed all = 44.646218558 MB/sec (with seq num) write speed all = 43.327987521 MB/sec (without seq num) 2024/11/19-15:22:33.437875 139851158521600 [compaction_job.cc:1948] [default] [JOB 169] Generated table #4973: 462086 keys, 55342040 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:33.437937 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000953437904, "cf_name": "default", "job": 169, "event": "table_file_creation", "file_number": 4973, "file_size": 55342040, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54462464, "index_size": 874112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12938408, "raw_average_key_size": 28, "raw_value_size": 53981292, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462086, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000953, "oldest_key_time": 0, "file_creation_time": 1732000953, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4973, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:33.451266 139851158521600 [compaction_job.cc:703] job-00169: subcompact[0], size: 0.110854 G, files: /storage/mytopling/.rocksdb [004972,004973] 2024/11/19-15:22:33.452476 139851158521600 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:22:33.452575 139851158521600 (Original Log Time 2024/11/19-15:22:32.715166) EVENT_LOG_v1 {"time_micros": 1732000952715150, "job": 169, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4872, 4874, 4876, 4880], "score": 5.72196, "input_data_size": 133300108, "oldest_snapshot_seqno": -1} 2024/11/19-15:22:33.452581 139851158521600 (Original Log Time 2024/11/19-15:22:33.452123) [compaction_job.cc:2022] [default] [JOB 169] Compacted 4@1 files to L2 => 110853808 bytes 2024/11/19-15:22:33.452584 139851158521600 (Original Log Time 2024/11/19-15:22:33.452445) [compaction_job.cc:1224] [default] compacted to: files[4 87 59 34 0 0 0] max score 3.92 (126 files need compaction), MB/sec: 181.1 rd, 150.6 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925636, records dropped: 0 output_compression: Snappy 2024/11/19-15:22:33.452599 139851158521600 (Original Log Time 2024/11/19-15:22:33.452470) EVENT_LOG_v1 {"time_micros": 1732000953452457, "cf": "default", "job": 169, "event": "compaction_finished", "compaction_time_micros": 736113, "compaction_time_cpu_micros": 711436, "output_level": 2, "num_output_files": 2, "total_output_size": 110853808, "num_input_records": 925636, "num_output_records": 925636, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 87, 59, 34, 0, 0, 0]} 2024/11/19-15:22:33.457359 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004820.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:33.457392 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000953457382, "job": 169, "event": "table_file_deletion", "file_number": 4820} 2024/11/19-15:22:33.462361 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004822.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:33.462389 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000953462381, "job": 169, "event": "table_file_deletion", "file_number": 4822} 2024/11/19-15:22:33.467720 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004824.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:33.467743 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000953467740, "job": 169, "event": "table_file_deletion", "file_number": 4824} 2024/11/19-15:22:33.472907 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004827.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:33.472935 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000953472933, "job": 169, "event": "table_file_deletion", "file_number": 4827} 2024/11/19-15:22:33.473103 139851158521600 [compaction_job.cc:2437] [default] [JOB 179] Compacting 4@1 files to L2, score 3.92, subcompactions 1 : 1 2024/11/19-15:22:33.473112 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 177 Base level 1, inputs: [4851(31MB) 4852(31MB) 4854(31MB) 4855(31MB)] 2024/11/19-15:22:33.496068 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-00 2024/11/19-15:22:33.496167 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00178/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:22:33.503841 139850978166528 [dcompact_etcd.cc:1163] job-00176/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.307 sec, input = 126.936 MiB 2024/11/19-15:22:33.785186 139851158521600 [dcompact_etcd.cc:1163] job-00179/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.312 sec, input = 126.810 MiB 2024/11/19-15:22:34.211630 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-02 2024/11/19-15:22:34.211707 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00170/att-02/compact.done) = No such file or directory: 11.229 sec on http probe failed: 5, giveup 2024/11/19-15:22:34.211725 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00170/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.513 (submit 275.452 ms wait 11.229 s), input 127.820 MiB, send shutdown 2024/11/19-15:22:34.219527 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00170/att-02 2024/11/19-15:22:34.219820 139851148035840 [dcompact_etcd.cc:1773] job-00170/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:34.221705 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00170) = success 2024/11/19-15:22:34.221713 139851148035840 [dcompact_etcd.cc:1773] job-00170/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:22:34.221732 139851148035840 (Original Log Time 2024/11/19-15:21:59.642106) EVENT_LOG_v1 {"time_micros": 1732000919642099, "job": 170, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4882, 4886, 4890, 4893], "score": 5.47232, "input_data_size": 133300044, "oldest_snapshot_seqno": -1} 2024/11/19-15:22:34.221740 139851148035840 [compaction_job.cc:2437] [default] [JOB 170] Compacting 4@1 files to L2, score 5.47, subcompactions 1 : 1 2024/11/19-15:22:34.221747 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 168 Base level 1, inputs: [4882(31MB) 4886(31MB) 4890(31MB) 4893(31MB)] 2024/11/19-15:22:34.221978 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004974.sst) /storage/mytopling/..//tmp/Topling-82038-BIa7e6 2024/11/19-15:22:34.366745 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004974.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:22:34.419524 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004974.sst): index pass time = 0.05's, 140.642'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463604 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:22:34.483507 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004974.sst): first pass time = 0.26's, 256.616'MB/sec 2024/11/19-15:22:34.483559 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004974.sst): old prealloc_size = 73819750, real_size = 55626752 2024/11/19-15:22:34.483790 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004974.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463604 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.75 avg-zval = 117.85 2024/11/19-15:22:34.483854 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004974.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.117 sec, Key+Value = 0.067 GB 2024/11/19-15:22:34.582456 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004974.sst): second pass time = 0.00's,175740272.727'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 561.453'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3885.112'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 2.442'MB/sec write SST data time = 0.05's, 1084.347'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 175740272.727'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463604 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.75 avg-zval = 117.85 usrkeys = 463604 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.75 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6615 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5734 value = 0.9907 dict = 0.00 all = 1.2088 } Zip/UnZip{ index = 0.0946 value = 1.0093 dict = 0.00 all = 0.8273 } ---------------------------- total value len = 9.604309 GB avg = 0.244 KB (by entry num) total key len = 1.020127 GB avg = 0.026 KB total ukey len = 0.704701 GB avg = 0.018 KB total ukey num = 0.039428261 Billion total entry num = 0.039428261 Billion write speed all = 44.712953612 MB/sec (with seq num) write speed all = 43.385482532 MB/sec (without seq num) 2024/11/19-15:22:34.590577 139851148035840 [compaction_job.cc:1948] [default] [JOB 170] Generated table #4974: 463604 keys, 55516120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:34.590643 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000954590612, "cf_name": "default", "job": 170, "event": "table_file_creation", "file_number": 4974, "file_size": 55516120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54633712, "index_size": 876928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12980912, "raw_average_key_size": 28, "raw_value_size": 54128004, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463604, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000954, "oldest_key_time": 0, "file_creation_time": 1732000954, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4974, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:34.605641 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004975.sst) /storage/mytopling/..//tmp/Topling-82038-avIMOX 2024/11/19-15:22:34.707007 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-02 2024/11/19-15:22:34.707104 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00172/att-02/compact.done) = No such file or directory: 9.215 sec on http probe failed: 4, retry 2024/11/19-15:22:34.767653 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004975.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:22:34.793032 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-00 2024/11/19-15:22:34.793136 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00177/att-00/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:22:34.801999 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004975.sst): index pass time = 0.03's, 215.448'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462041 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:22:34.870398 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004975.sst): first pass time = 0.26's, 252.779'MB/sec 2024/11/19-15:22:34.870452 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004975.sst): old prealloc_size = 73819750, real_size = 55454688 2024/11/19-15:22:34.870684 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004975.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462041 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.84 avg-zval = 117.87 2024/11/19-15:22:34.870746 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004975.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.103 sec, Key+Value = 0.067 GB 2024/11/19-15:22:34.966062 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004975.sst): second pass time = 0.00's,163091317.221'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 578.946'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3842.940'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.569'MB/sec write SST data time = 0.05's, 1097.870'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 163091317.221'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462041 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.84 avg-zval = 117.87 usrkeys = 462041 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.84 avg-zval = 117.87 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6597 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0553 GB } UnZip/Zip{ index = 10.5002 value = 0.9913 dict = 0.00 all = 1.2092 } Zip/UnZip{ index = 0.0952 value = 1.0088 dict = 0.00 all = 0.8270 } ---------------------------- total value len = 9.658292 GB avg = 0.242 KB (by entry num) total key len = 1.033065 GB avg = 0.026 KB total ukey len = 0.713942 GB avg = 0.018 KB total ukey num = 0.039890302 Billion total entry num = 0.039890302 Billion write speed all = 44.922065321 MB/sec (with seq num) write speed all = 43.581202946 MB/sec (without seq num) 2024/11/19-15:22:34.969838 139851148035840 [compaction_job.cc:1948] [default] [JOB 170] Generated table #4975: 462041 keys, 55344536 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:34.969888 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000954969865, "cf_name": "default", "job": 170, "event": "table_file_creation", "file_number": 4975, "file_size": 55344536, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54458960, "index_size": 880064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12937148, "raw_average_key_size": 28, "raw_value_size": 53983226, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462041, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000954, "oldest_key_time": 0, "file_creation_time": 1732000954, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4975, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:34.984796 139851148035840 [compaction_job.cc:703] job-00170: subcompact[0], size: 0.110861 G, files: /storage/mytopling/.rocksdb [004974,004975] 2024/11/19-15:22:34.986102 139851148035840 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:22:34.986202 139851148035840 (Original Log Time 2024/11/19-15:22:34.221777) EVENT_LOG_v1 {"time_micros": 1732000954221759, "job": 170, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4882, 4886, 4890, 4893], "score": 5.47232, "input_data_size": 133300044, "oldest_snapshot_seqno": -1} 2024/11/19-15:22:34.986209 139851148035840 (Original Log Time 2024/11/19-15:22:34.985739) [compaction_job.cc:2022] [default] [JOB 170] Compacted 4@1 files to L2 => 110860656 bytes 2024/11/19-15:22:34.986212 139851148035840 (Original Log Time 2024/11/19-15:22:34.986070) [compaction_job.cc:1224] [default] compacted to: files[4 83 61 34 0 0 0] max score 3.67 (124 files need compaction), MB/sec: 174.7 rd, 145.3 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925645, records dropped: 0 output_compression: Snappy 2024/11/19-15:22:34.986229 139851148035840 (Original Log Time 2024/11/19-15:22:34.986096) EVENT_LOG_v1 {"time_micros": 1732000954986083, "cf": "default", "job": 170, "event": "compaction_finished", "compaction_time_micros": 763032, "compaction_time_cpu_micros": 750963, "output_level": 2, "num_output_files": 2, "total_output_size": 110860656, "num_input_records": 925645, "num_output_records": 925645, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 83, 61, 34, 0, 0, 0]} 2024/11/19-15:22:34.990967 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004838.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:34.990993 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000954990990, "job": 170, "event": "table_file_deletion", "file_number": 4838} 2024/11/19-15:22:34.995775 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004840.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:34.995798 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000954995795, "job": 170, "event": "table_file_deletion", "file_number": 4840} 2024/11/19-15:22:34.996234 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004843.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:34.996247 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000954996246, "job": 170, "event": "table_file_deletion", "file_number": 4843} 2024/11/19-15:22:34.998596 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004844.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:34.998615 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000954998613, "job": 170, "event": "table_file_deletion", "file_number": 4844} 2024/11/19-15:22:34.998758 139851148035840 [compaction_job.cc:2437] [default] [JOB 180] Compacting 4@1 files to L2, score 3.67, subcompactions 1 : 1 2024/11/19-15:22:34.998767 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 178 Base level 1, inputs: [4856(31MB) 4858(31MB) 4859(31MB) 4860(31MB)] 2024/11/19-15:22:35.005468 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4976. Immutable memtables: 0. 2024/11/19-15:22:35.005639 139852362282752 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:22:35.005867 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 181] Syncing log #4965 2024/11/19-15:22:35.005982 139850967680768 (Original Log Time 2024/11/19-15:22:35.005679) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:22:35.005990 139850967680768 [flush_job.cc:879] [default] [JOB 181] Flushing memtable with next log file: 4976 2024/11/19-15:22:35.006004 139850967680768 EVENT_LOG_v1 {"time_micros": 1732000955006000, "job": 181, "event": "flush_started", "num_memtables": 1, "num_entries": 1356038, "num_deletes": 0, "total_data_size": 467074735, "memory_usage": 538705288, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:22:35.006008 139850967680768 [flush_job.cc:910] [default] [JOB 181] Level-0 flush table #4977: started 2024/11/19-15:22:35.006458 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000022.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004977.sst in ConvertToSST, mem_size = 513.749 M, time = 441.797 us 2024/11/19-15:22:35.253694 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-02 2024/11/19-15:22:35.253795 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00174/att-02/compact.done) = No such file or directory: 4.379 sec on http probe failed: 2, retry 2024/11/19-15:22:35.307906 139851148035840 [dcompact_etcd.cc:1163] job-00180/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.309 sec, input = 126.844 MiB 2024/11/19-15:22:35.501559 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-00 2024/11/19-15:22:35.501713 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00178/att-00/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:22:35.541217 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004977.sst): fsize = 513.751 M, time(ms): open: 0.030, seek: 0.014, finish: 0.033, meta: 0.010, Flush: 0.012, sync: 534.637, close: 0.009, all: 534.745 2024/11/19-15:22:35.542277 139850967680768 [flush_job.cc:1083] [default] [JOB 181] Flush lasted 536294 microseconds, and 385059 cpu microseconds. 2024/11/19-15:22:35.562513 139850967680768 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:22:35.562608 139850967680768 (Original Log Time 2024/11/19-15:22:35.541261) [flush_job.cc:1029] [default] [JOB 181] Level-0 flush table #4977: 538706714 bytes OK (needs compaction) 2024/11/19-15:22:35.562613 139850967680768 (Original Log Time 2024/11/19-15:22:35.542293) [memtable_list.cc:560] [default] Level-0 commit table #4977 started 2024/11/19-15:22:35.562617 139850967680768 (Original Log Time 2024/11/19-15:22:35.562480) [memtable_list.cc:759] [default] Level-0 commit table #4977: memtable #1 done 2024/11/19-15:22:35.562619 139850967680768 (Original Log Time 2024/11/19-15:22:35.562506) EVENT_LOG_v1 {"time_micros": 1732000955562500, "job": 181, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [5, 83, 61, 34, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:22:35.562622 139850967680768 (Original Log Time 2024/11/19-15:22:35.562580) [db_impl_compaction_flush.cc:310] [default] Level summary: files[5 83 61 34 0 0 0] max score 3.42 (121 files need compaction) 2024/11/19-15:22:36.686146 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-01 2024/11/19-15:22:36.686238 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00176/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:22:36.716231 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-02 2024/11/19-15:22:36.716302 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00172/att-02/compact.done) = No such file or directory: 11.223 sec on http probe failed: 5, giveup 2024/11/19-15:22:36.716325 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00172/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.821 sec, real 11.530 (submit 297.441 ms wait 11.223 s), input 127.821 MiB, send shutdown 2024/11/19-15:22:36.724368 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00172/att-02 2024/11/19-15:22:36.724618 139851137550080 [dcompact_etcd.cc:1773] job-00172/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:36.726561 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00172) = success 2024/11/19-15:22:36.726574 139851137550080 [dcompact_etcd.cc:1773] job-00172/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:22:36.726600 139851137550080 (Original Log Time 2024/11/19-15:22:02.079974) EVENT_LOG_v1 {"time_micros": 1732000922079955, "job": 172, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4896, 4899, 4905, 4908], "score": 5.22267, "input_data_size": 133300172, "oldest_snapshot_seqno": -1} 2024/11/19-15:22:36.726611 139851137550080 [compaction_job.cc:2437] [default] [JOB 172] Compacting 4@1 files to L2, score 5.22, subcompactions 1 : 1 2024/11/19-15:22:36.726622 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 169 Base level 1, inputs: [4896(31MB) 4899(31MB) 4905(31MB) 4908(31MB)] 2024/11/19-15:22:36.726911 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004978.sst) /storage/mytopling/..//tmp/Topling-82038-F66tDm 2024/11/19-15:22:36.801965 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-00 2024/11/19-15:22:36.802070 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00177/att-00/compact.done) = No such file or directory: 7.192 sec on http probe failed: 3, retry 2024/11/19-15:22:36.887905 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004978.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:22:36.922392 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004978.sst): index pass time = 0.03's, 215.312'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463520 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:22:36.960634 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-00 2024/11/19-15:22:36.960764 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00179/att-00/compact.done) = No such file or directory: 3.170 sec on http probe failed: 1, retry 2024/11/19-15:22:36.990053 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004978.sst): first pass time = 0.26's, 255.058'MB/sec 2024/11/19-15:22:36.990105 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004978.sst): old prealloc_size = 73819750, real_size = 55619664 2024/11/19-15:22:36.990356 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004978.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463520 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.78 avg-zval = 117.85 2024/11/19-15:22:36.990424 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004978.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.103 sec, Key+Value = 0.067 GB 2024/11/19-15:22:37.083405 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004978.sst): second pass time = 0.00's,133986168.317'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 595.034'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3515.959'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.644'MB/sec write SST data time = 0.05's, 1128.034'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 133986168.317'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463520 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 117.85 usrkeys = 463520 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6614 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5738 value = 0.9909 dict = 0.00 all = 1.2090 } Zip/UnZip{ index = 0.0946 value = 1.0092 dict = 0.00 all = 0.8271 } ---------------------------- total value len = 9.712423 GB avg = 0.241 KB (by entry num) total key len = 1.046043 GB avg = 0.026 KB total ukey len = 0.723213 GB avg = 0.018 KB total ukey num = 0.040353822 Billion total entry num = 0.040353822 Billion write speed all = 44.805427497 MB/sec (with seq num) write speed all = 43.460945642 MB/sec (without seq num) 2024/11/19-15:22:37.086888 139851137550080 [compaction_job.cc:1948] [default] [JOB 172] Generated table #4978: 463520 keys, 55509080 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:37.086948 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000957086916, "cf_name": "default", "job": 172, "event": "table_file_creation", "file_number": 4978, "file_size": 55509080, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54626880, "index_size": 876736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12978560, "raw_average_key_size": 28, "raw_value_size": 54130412, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463520, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000956, "oldest_key_time": 0, "file_creation_time": 1732000956, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4978, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:37.100966 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004979.sst) /storage/mytopling/..//tmp/Topling-82038-mXVoSU 2024/11/19-15:22:37.201189 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-02 2024/11/19-15:22:37.201304 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00174/att-02/compact.done) = No such file or directory: 6.331 sec on http probe failed: 3, retry 2024/11/19-15:22:37.242252 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004979.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:22:37.275856 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004979.sst): index pass time = 0.03's, 220.246'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462143 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:22:37.343629 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004979.sst): first pass time = 0.24's, 275.798'MB/sec 2024/11/19-15:22:37.343683 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004979.sst): old prealloc_size = 73819750, real_size = 55457520 2024/11/19-15:22:37.343937 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004979.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462143 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.81 avg-zval = 117.86 2024/11/19-15:22:37.344004 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004979.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.102 sec, Key+Value = 0.067 GB 2024/11/19-15:22:37.437741 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004979.sst): second pass time = 0.00's,156015589.595'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 588.539'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3454.074'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.598'MB/sec write SST data time = 0.05's, 1122.330'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 156015589.595'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462143 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 117.86 usrkeys = 462143 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6597 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0553 GB } UnZip/Zip{ index = 10.5725 value = 0.9911 dict = 0.00 all = 1.2091 } Zip/UnZip{ index = 0.0946 value = 1.0090 dict = 0.00 all = 0.8270 } ---------------------------- total value len = 9.766404 GB avg = 0.239 KB (by entry num) total key len = 1.058983 GB avg = 0.026 KB total ukey len = 0.732455 GB avg = 0.018 KB total ukey num = 0.040815965 Billion total entry num = 0.040815965 Billion write speed all = 45.017700816 MB/sec (with seq num) write speed all = 43.659825385 MB/sec (without seq num) 2024/11/19-15:22:37.441410 139851137550080 [compaction_job.cc:1948] [default] [JOB 172] Generated table #4979: 462143 keys, 55347288 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:37.441461 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000957441439, "cf_name": "default", "job": 172, "event": "table_file_creation", "file_number": 4979, "file_size": 55347288, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54467552, "index_size": 874240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12940004, "raw_average_key_size": 28, "raw_value_size": 53981394, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462143, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732000957, "oldest_key_time": 0, "file_creation_time": 1732000957, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4979, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:37.455482 139851137550080 [compaction_job.cc:703] job-00172: subcompact[0], size: 0.110856 G, files: /storage/mytopling/.rocksdb [004978,004979] 2024/11/19-15:22:37.456728 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:22:37.456903 139851137550080 (Original Log Time 2024/11/19-15:22:36.726662) EVENT_LOG_v1 {"time_micros": 1732000956726638, "job": 172, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4896, 4899, 4905, 4908], "score": 5.22267, "input_data_size": 133300172, "oldest_snapshot_seqno": -1} 2024/11/19-15:22:37.456913 139851137550080 (Original Log Time 2024/11/19-15:22:37.456389) [compaction_job.cc:2022] [default] [JOB 172] Compacted 4@1 files to L2 => 110856368 bytes 2024/11/19-15:22:37.456918 139851137550080 (Original Log Time 2024/11/19-15:22:37.456697) [compaction_job.cc:1224] [default] compacted to: files[5 79 63 34 0 0 0] max score 3.42 (123 files need compaction), MB/sec: 182.9 rd, 152.1 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925663, records dropped: 0 output_compression: Snappy 2024/11/19-15:22:37.456953 139851137550080 (Original Log Time 2024/11/19-15:22:37.456721) EVENT_LOG_v1 {"time_micros": 1732000957456708, "cf": "default", "job": 172, "event": "compaction_finished", "compaction_time_micros": 728834, "compaction_time_cpu_micros": 725803, "output_level": 2, "num_output_files": 2, "total_output_size": 110856368, "num_input_records": 925663, "num_output_records": 925663, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 79, 63, 34, 0, 0, 0]} 2024/11/19-15:22:37.459762 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004817.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:37.459786 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000957459783, "job": 172, "event": "table_file_deletion", "file_number": 4817} 2024/11/19-15:22:37.460456 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004826.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:37.460471 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000957460469, "job": 172, "event": "table_file_deletion", "file_number": 4826} 2024/11/19-15:22:37.464783 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004829.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:37.464804 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000957464802, "job": 172, "event": "table_file_deletion", "file_number": 4829} 2024/11/19-15:22:37.467768 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004831.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:37.467787 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000957467785, "job": 172, "event": "table_file_deletion", "file_number": 4831} 2024/11/19-15:22:37.472259 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004833.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:37.472278 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000957472276, "job": 172, "event": "table_file_deletion", "file_number": 4833} 2024/11/19-15:22:37.476941 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004834.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:37.476961 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000957476959, "job": 172, "event": "table_file_deletion", "file_number": 4834} 2024/11/19-15:22:37.481730 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004836.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:37.481749 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000957481747, "job": 172, "event": "table_file_deletion", "file_number": 4836} 2024/11/19-15:22:37.481898 139851137550080 [compaction_job.cc:2437] [default] [JOB 182] Compacting 4@1 files to L2, score 3.42, subcompactions 1 : 1 2024/11/19-15:22:37.481907 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 180 Base level 1, inputs: [4862(31MB) 4863(31MB) 4864(31MB) 4866(31MB)] 2024/11/19-15:22:37.510764 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-00 2024/11/19-15:22:37.510852 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00178/att-00/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:22:37.796768 139851137550080 [dcompact_etcd.cc:1163] job-00182/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.314 sec, input = 126.864 MiB 2024/11/19-15:22:38.485515 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-00 2024/11/19-15:22:38.485651 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00180/att-00/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:22:38.691453 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-01 2024/11/19-15:22:38.691529 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00176/att-01/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:22:38.810455 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-00 2024/11/19-15:22:38.810554 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00177/att-00/compact.done) = No such file or directory: 9.200 sec on http probe failed: 4, retry 2024/11/19-15:22:38.969575 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-00 2024/11/19-15:22:38.969663 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00179/att-00/compact.done) = No such file or directory: 5.176 sec on http probe failed: 2, retry 2024/11/19-15:22:39.153338 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-02 2024/11/19-15:22:39.153449 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00174/att-02/compact.done) = No such file or directory: 8.279 sec on http probe failed: 4, retry 2024/11/19-15:22:39.519638 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-00 2024/11/19-15:22:39.519734 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00178/att-00/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:22:40.490931 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-00 2024/11/19-15:22:40.491012 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00180/att-00/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:22:40.700042 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-01 2024/11/19-15:22:40.700119 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00176/att-01/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:22:40.816275 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-00 2024/11/19-15:22:40.816353 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00177/att-00/compact.done) = No such file or directory: 11.209 sec on http probe failed: 5, giveup 2024/11/19-15:22:40.816371 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00177/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.970 sec, real 11.524 (submit 309.587 ms wait 11.209 s), input 126.970 MiB, send shutdown 2024/11/19-15:22:40.824391 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-00 2024/11/19-15:22:40.824587 139851169007360 [dcompact_etcd.cc:1773] job-00177/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:40.974794 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-00 2024/11/19-15:22:40.974883 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00179/att-00/compact.done) = No such file or directory: 7.185 sec on http probe failed: 3, retry 2024/11/19-15:22:40.977285 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-00 2024/11/19-15:22:40.977367 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00182/att-00/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:22:41.104420 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-02 2024/11/19-15:22:41.104490 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00174/att-02/compact.done) = No such file or directory: 10.231 sec on http probe failed: 5, giveup 2024/11/19-15:22:41.104508 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00174/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 97.103 sec, real 10.534 (submit 294.147 ms wait 10.231 s), input 97.103 MiB, send shutdown 2024/11/19-15:22:41.113478 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00174/att-02 2024/11/19-15:22:41.113686 139851179493120 [dcompact_etcd.cc:1773] job-00174/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:41.115546 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00174) = success 2024/11/19-15:22:41.115560 139851179493120 [dcompact_etcd.cc:1773] job-00174/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:22:41.115581 139851179493120 (Original Log Time 2024/11/19-15:22:09.515538) EVENT_LOG_v1 {"time_micros": 1732000929515531, "job": 174, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4920, 4818, 4821, 4823], "score": 4.86227, "input_data_size": 102674044, "oldest_snapshot_seqno": -1} 2024/11/19-15:22:41.115588 139851179493120 [compaction_job.cc:2437] [default] [JOB 174] Compacting 4@1 files to L2, score 4.86, subcompactions 1 : 1 2024/11/19-15:22:41.115596 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 172 Base level 1, inputs: [4920(2051KB) 4818(31MB) 4821(31MB) 4823(31MB)] 2024/11/19-15:22:41.115801 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004980.sst) /storage/mytopling/..//tmp/Topling-82038-6VSgFM 2024/11/19-15:22:41.133179 139851169007360 [dcompact_etcd.cc:1163] job-00177/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.309 sec, input = 126.970 MiB 2024/11/19-15:22:41.170142 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004980.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:22:41.170245 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004980.sst): index pass time = 0.00's,6390.599'MB/sec index type = UintIndex_AllOne usrkeys = 64716 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:22:41.236619 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004980.sst): first pass time = 0.12's, 555.510'MB/sec 2024/11/19-15:22:41.236666 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004980.sst): old prealloc_size = 73819750, real_size = 66041736 2024/11/19-15:22:41.236689 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004980.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64716 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.98 avg-zval =1020.23 2024/11/19-15:22:41.290766 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004980.sst): second pass time = 0.00's,169625484.536'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1220.860'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.600'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.422'MB/sec write SST data time = 0.02's, 4189.178'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 169625484.536'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64716 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.98 avg-zval = 1020.23 usrkeys = 64716 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.98 avg-zval = 1020.23 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.7500 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 9.832219 GB avg = 0.241 KB (by entry num) total key len = 1.060278 GB avg = 0.026 KB total ukey len = 0.733232 GB avg = 0.018 KB total ukey num = 0.040880681 Billion total entry num = 0.040880681 Billion write speed all = 44.582434833 MB/sec (with seq num) write speed all = 43.243854384 MB/sec (without seq num) 2024/11/19-15:22:41.294318 139851179493120 [compaction_job.cc:1948] [default] [JOB 174] Generated table #4980: 64716 keys, 66026570 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:41.294372 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000961294348, "cf_name": "default", "job": 174, "event": "table_file_creation", "file_number": 4980, "file_size": 66026570, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025232, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294320, "raw_average_key_size": 20, "raw_value_size": 65814688, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64716, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000961, "oldest_key_time": 0, "file_creation_time": 1732000961, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4980, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:41.309222 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004981.sst) /storage/mytopling/..//tmp/Topling-82038-ocftCa 2024/11/19-15:22:41.336300 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004981.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0002 GB, waited 0.000 sec, Key+Value = 0.035 GB 2024/11/19-15:22:41.338106 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004981.sst): index pass time = 0.00's, 150.347'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 33471 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0003 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.73 2024/11/19-15:22:41.370577 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004981.sst): first pass time = 0.06's, 565.941'MB/sec 2024/11/19-15:22:41.370623 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004981.sst): old prealloc_size = 73819750, real_size = 34216576 2024/11/19-15:22:41.370666 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004981.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 33471 prefix = 4 raw-key = 0.0003 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.73 raw-val = 0.0340 GB zip-val = 0.0342 GB avg-val =1017.04 avg-zval =1020.29 2024/11/19-15:22:41.370705 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004981.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.034 sec, Key+Value = 0.035 GB 2024/11/19-15:22:41.401687 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004981.sst): second pass time = 0.00's,65971412.791'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.03's, 1101.820'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1492.331'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 0.378'MB/sec write SST data time = 0.01's, 3852.289'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.034'GB zip my value throughput = 65971412.791'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 33471 avg-key = 20.00 avg-zkey = 1.73 avg-val = 1017.04 avg-zval = 1020.29 usrkeys = 33471 avg-key = 12.00 avg-zkey = 1.73 avg-val = 1017.04 avg-zval = 1020.29 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 4173 UnZipSize{ index = 0.0004 GB value = 0.0340 GB dict = 0.00 MB all = 0.0347 GB } __ZipSize{ index = 0.0001 GB value = 0.0342 GB dict = 0.00 MB all = 0.0342 GB } UnZip/Zip{ index = 6.9500 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1439 value = 1.0032 dict = 0.00 all = 0.9856 } ---------------------------- total value len = 9.866260 GB avg = 0.241 KB (by entry num) total key len = 1.060947 GB avg = 0.026 KB total ukey len = 0.733634 GB avg = 0.018 KB total ukey num = 0.040914152 Billion total entry num = 0.040914152 Billion write speed all = 44.704207950 MB/sec (with seq num) write speed all = 43.365139482 MB/sec (without seq num) 2024/11/19-15:22:41.403862 139851179493120 [compaction_job.cc:1948] [default] [JOB 174] Generated table #4981: 33471 keys, 34209476 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:41.403910 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000961403887, "cf_name": "default", "job": 174, "event": "table_file_creation", "file_number": 4981, "file_size": 34209476, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34150256, "index_size": 57792, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 669420, "raw_average_key_size": 20, "raw_value_size": 34041249, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 33471, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000961, "oldest_key_time": 0, "file_creation_time": 1732000961, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4981, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:41.411640 139851179493120 [compaction_job.cc:703] job-00174: subcompact[0], size: 0.100236 G, files: /storage/mytopling/.rocksdb [004980,004981] 2024/11/19-15:22:41.412339 139851179493120 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777216 2024/11/19-15:22:41.412472 139851179493120 (Original Log Time 2024/11/19-15:22:41.115625) EVENT_LOG_v1 {"time_micros": 1732000961115607, "job": 174, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4920, 4818, 4821, 4823], "score": 4.86227, "input_data_size": 102674044, "oldest_snapshot_seqno": -1} 2024/11/19-15:22:41.412478 139851179493120 (Original Log Time 2024/11/19-15:22:41.412003) [compaction_job.cc:2022] [default] [JOB 174] Compacted 4@1 files to L2 => 100236046 bytes 2024/11/19-15:22:41.412481 139851179493120 (Original Log Time 2024/11/19-15:22:41.412310) [compaction_job.cc:1224] [default] compacted to: files[5 75 65 34 0 0 0] max score 3.17 (121 files need compaction), MB/sec: 346.8 rd, 338.6 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(97.9, 0.0 +0.0 blob) out(95.6 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 98187, records dropped: 0 output_compression: Snappy 2024/11/19-15:22:41.412498 139851179493120 (Original Log Time 2024/11/19-15:22:41.412333) EVENT_LOG_v1 {"time_micros": 1732000961412320, "cf": "default", "job": 174, "event": "compaction_finished", "compaction_time_micros": 296027, "compaction_time_cpu_micros": 294538, "output_level": 2, "num_output_files": 2, "total_output_size": 100236046, "num_input_records": 98187, "num_output_records": 98187, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 75, 65, 34, 0, 0, 0]} 2024/11/19-15:22:41.416260 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004912.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:41.416282 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000961416279, "job": 174, "event": "table_file_deletion", "file_number": 4912} 2024/11/19-15:22:41.419200 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004916.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:41.419219 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000961419217, "job": 174, "event": "table_file_deletion", "file_number": 4916} 2024/11/19-15:22:41.419364 139851179493120 [compaction_job.cc:2437] [default] [JOB 183] Compacting 4@1 files to L2, score 3.17, subcompactions 1 : 1 2024/11/19-15:22:41.419373 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 181 Base level 1, inputs: [4867(31MB) 4868(31MB) 4870(31MB) 4871(31MB)] 2024/11/19-15:22:41.527686 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-00 2024/11/19-15:22:41.527776 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00178/att-00/compact.done) = No such file or directory: 11.206 sec on http probe failed: 5, giveup 2024/11/19-15:22:41.527794 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00178/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.907 sec, real 11.491 (submit 276.691 ms wait 11.206 s), input 126.907 MiB, send shutdown 2024/11/19-15:22:41.535511 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-00 2024/11/19-15:22:41.535729 139851127064320 [dcompact_etcd.cc:1773] job-00178/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:41.714731 139851179493120 [dcompact_etcd.cc:1163] job-00183/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.295 sec, input = 126.966 MiB 2024/11/19-15:22:41.783561 139851127064320 [dcompact_etcd.cc:1163] job-00178/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.248 sec, input = 126.907 MiB 2024/11/19-15:22:42.499390 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-00 2024/11/19-15:22:42.499468 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00180/att-00/compact.done) = No such file or directory: 7.183 sec on http probe failed: 3, retry 2024/11/19-15:22:42.709746 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-01 2024/11/19-15:22:42.709835 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00176/att-01/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:22:42.979870 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-00 2024/11/19-15:22:42.979966 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00179/att-00/compact.done) = No such file or directory: 9.190 sec on http probe failed: 4, retry 2024/11/19-15:22:42.982156 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-00 2024/11/19-15:22:42.982221 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00182/att-00/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:22:44.316327 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-01 2024/11/19-15:22:44.316417 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00177/att-01/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:22:44.509499 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-00 2024/11/19-15:22:44.509564 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00180/att-00/compact.done) = No such file or directory: 9.192 sec on http probe failed: 4, retry 2024/11/19-15:22:44.719005 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-01 2024/11/19-15:22:44.719063 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00176/att-01/compact.done) = No such file or directory: 11.206 sec on http probe failed: 5, giveup 2024/11/19-15:22:44.719080 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00176/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.936 sec, real 11.522 (submit 306.862 ms wait 11.206 s), input 126.936 MiB, send shutdown 2024/11/19-15:22:44.727330 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-01 2024/11/19-15:22:44.727544 139850978166528 [dcompact_etcd.cc:1773] job-00176/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:44.897761 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-00 2024/11/19-15:22:44.897828 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00183/att-00/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:22:44.962152 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-01 2024/11/19-15:22:44.962217 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00178/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:22:44.987960 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-00 2024/11/19-15:22:44.988014 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00179/att-00/compact.done) = No such file or directory: 11.195 sec on http probe failed: 5, giveup 2024/11/19-15:22:44.988029 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00179/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.810 sec, real 11.514 (submit 311.614 ms wait 11.195 s), input 126.810 MiB, send shutdown 2024/11/19-15:22:44.990073 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-00 2024/11/19-15:22:44.990123 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00182/att-00/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:22:44.996544 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-00 2024/11/19-15:22:44.996750 139851158521600 [dcompact_etcd.cc:1773] job-00179/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:45.028404 139850978166528 [dcompact_etcd.cc:1163] job-00176/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.301 sec, input = 126.936 MiB 2024/11/19-15:22:45.272305 139851158521600 [dcompact_etcd.cc:1163] job-00179/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.275 sec, input = 126.810 MiB 2024/11/19-15:22:46.325185 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-01 2024/11/19-15:22:46.325264 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00177/att-01/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:22:46.515011 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-00 2024/11/19-15:22:46.515073 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00180/att-00/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:22:46.515090 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00180/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.844 sec, real 11.516 (submit 308.674 ms wait 11.202 s), input 126.844 MiB, send shutdown 2024/11/19-15:22:46.519453 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-00 2024/11/19-15:22:46.519674 139851148035840 [dcompact_etcd.cc:1773] job-00180/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:46.787379 139851148035840 [dcompact_etcd.cc:1163] job-00180/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.268 sec, input = 126.844 MiB 2024/11/19-15:22:46.902899 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-00 2024/11/19-15:22:46.902988 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00183/att-00/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:22:46.970625 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-01 2024/11/19-15:22:46.970692 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00178/att-01/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:22:46.998099 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-00 2024/11/19-15:22:46.998154 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00182/att-00/compact.done) = No such file or directory: 9.193 sec on http probe failed: 4, retry 2024/11/19-15:22:48.212471 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-02 2024/11/19-15:22:48.212554 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00176/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:22:48.330021 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-01 2024/11/19-15:22:48.330125 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00177/att-01/compact.done) = No such file or directory: 7.192 sec on http probe failed: 3, retry 2024/11/19-15:22:48.452361 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-01 2024/11/19-15:22:48.452439 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00179/att-01/compact.done) = No such file or directory: 3.170 sec on http probe failed: 1, retry 2024/11/19-15:22:48.907900 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-00 2024/11/19-15:22:48.908001 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00183/att-00/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:22:48.980622 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-01 2024/11/19-15:22:48.980709 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00178/att-01/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:22:49.006126 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-00 2024/11/19-15:22:49.006187 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00182/att-00/compact.done) = No such file or directory: 11.201 sec on http probe failed: 5, giveup 2024/11/19-15:22:49.006204 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00182/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.864 sec, real 11.524 (submit 314.402 ms wait 11.201 s), input 126.864 MiB, send shutdown 2024/11/19-15:22:49.014656 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-00 2024/11/19-15:22:49.014883 139851137550080 [dcompact_etcd.cc:1773] job-00182/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:49.300433 139851137550080 [dcompact_etcd.cc:1163] job-00182/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.285 sec, input = 126.864 MiB 2024/11/19-15:22:49.407071 139852362282752 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16273899 2024/11/19-15:22:49.966780 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-01 2024/11/19-15:22:49.966857 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00180/att-01/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:22:50.221111 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-02 2024/11/19-15:22:50.221201 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00176/att-02/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:22:50.339400 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-01 2024/11/19-15:22:50.339468 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00177/att-01/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:22:50.457249 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-01 2024/11/19-15:22:50.457305 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00179/att-01/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:22:50.914179 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-00 2024/11/19-15:22:50.914244 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00183/att-00/compact.done) = No such file or directory: 9.193 sec on http probe failed: 4, retry 2024/11/19-15:22:50.989391 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-01 2024/11/19-15:22:50.989448 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00178/att-01/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:22:51.971839 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-01 2024/11/19-15:22:51.971916 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00180/att-01/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:22:52.229389 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-02 2024/11/19-15:22:52.229461 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00176/att-02/compact.done) = No such file or directory: 7.193 sec on http probe failed: 3, retry 2024/11/19-15:22:52.348575 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-01 2024/11/19-15:22:52.348652 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00177/att-01/compact.done) = No such file or directory: 11.206 sec on http probe failed: 5, giveup 2024/11/19-15:22:52.348671 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00177/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.970 sec, real 11.524 (submit 308.523 ms wait 11.206 s), input 126.970 MiB, send shutdown 2024/11/19-15:22:52.353101 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-01 2024/11/19-15:22:52.353315 139851169007360 [dcompact_etcd.cc:1773] job-00177/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:52.466419 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-01 2024/11/19-15:22:52.466503 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00179/att-01/compact.done) = No such file or directory: 7.185 sec on http probe failed: 3, retry 2024/11/19-15:22:52.481632 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-01 2024/11/19-15:22:52.481710 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00182/att-01/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:22:52.645835 139851169007360 [dcompact_etcd.cc:1163] job-00177/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.292 sec, input = 126.970 MiB 2024/11/19-15:22:52.922926 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-00 2024/11/19-15:22:52.922994 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00183/att-00/compact.done) = No such file or directory: 11.200 sec on http probe failed: 5, giveup 2024/11/19-15:22:52.923023 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00183/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.966 sec, real 11.503 (submit 294.952 ms wait 11.200 s), input 126.966 MiB, send shutdown 2024/11/19-15:22:52.928460 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-00 2024/11/19-15:22:52.928672 139851179493120 [dcompact_etcd.cc:1773] job-00183/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:52.994903 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-01 2024/11/19-15:22:52.994965 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00178/att-01/compact.done) = No such file or directory: 11.206 sec on http probe failed: 5, giveup 2024/11/19-15:22:52.994980 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00178/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.907 sec, real 11.459 (submit 247.766 ms wait 11.206 s), input 126.907 MiB, send shutdown 2024/11/19-15:22:52.999278 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-01 2024/11/19-15:22:52.999456 139851127064320 [dcompact_etcd.cc:1773] job-00178/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:53.177054 139851179493120 [dcompact_etcd.cc:1163] job-00183/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.248 sec, input = 126.966 MiB 2024/11/19-15:22:53.283879 139851127064320 [dcompact_etcd.cc:1163] job-00178/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.284 sec, input = 126.907 MiB 2024/11/19-15:22:53.978196 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-01 2024/11/19-15:22:53.978288 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00180/att-01/compact.done) = No such file or directory: 7.185 sec on http probe failed: 3, retry 2024/11/19-15:22:54.236088 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-02 2024/11/19-15:22:54.236161 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00176/att-02/compact.done) = No such file or directory: 9.201 sec on http probe failed: 4, retry 2024/11/19-15:22:54.474785 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-01 2024/11/19-15:22:54.474874 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00179/att-01/compact.done) = No such file or directory: 9.194 sec on http probe failed: 4, retry 2024/11/19-15:22:54.486738 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-01 2024/11/19-15:22:54.486802 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00182/att-01/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:22:55.829008 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-02 2024/11/19-15:22:55.829112 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00177/att-02/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:22:55.983862 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-01 2024/11/19-15:22:55.983971 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00180/att-01/compact.done) = No such file or directory: 9.191 sec on http probe failed: 4, retry 2024/11/19-15:22:56.240901 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-02 2024/11/19-15:22:56.241005 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00176/att-02/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:22:56.241024 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00176/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.936 sec, real 11.513 (submit 300.795 ms wait 11.208 s), input 126.936 MiB, send shutdown 2024/11/19-15:22:56.246293 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00176/att-02 2024/11/19-15:22:56.246503 139850978166528 [dcompact_etcd.cc:1773] job-00176/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:56.248406 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00176) = success 2024/11/19-15:22:56.248413 139850978166528 [dcompact_etcd.cc:1773] job-00176/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:22:56.248431 139850978166528 (Original Log Time 2024/11/19-15:22:21.633230) EVENT_LOG_v1 {"time_micros": 1732000941633224, "job": 176, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4825, 4828, 4830, 4832], "score": 4.67102, "input_data_size": 134098880, "oldest_snapshot_seqno": -1} 2024/11/19-15:22:56.248439 139850978166528 [compaction_job.cc:2437] [default] [JOB 176] Compacting 4@1 files to L2, score 4.67, subcompactions 1 : 1 2024/11/19-15:22:56.248448 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 174 Base level 1, inputs: [4825(31MB) 4828(31MB) 4830(31MB) 4832(31MB)] 2024/11/19-15:22:56.248670 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004982.sst) /storage/mytopling/..//tmp/Topling-82038-hi6E3r 2024/11/19-15:22:56.300761 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004982.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:22:56.300878 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004982.sst): index pass time = 0.00's,6007.101'MB/sec index type = UintIndex_AllOne usrkeys = 64716 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 6 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:22:56.360993 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-01 2024/11/19-15:22:56.361105 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00183/att-01/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:22:56.366829 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004982.sst): first pass time = 0.12's, 568.021'MB/sec 2024/11/19-15:22:56.366876 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004982.sst): old prealloc_size = 73819750, real_size = 66042360 2024/11/19-15:22:56.366898 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004982.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64716 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.99 avg-zval =1020.24 2024/11/19-15:22:56.420773 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004982.sst): second pass time = 0.00's,206966342.767'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1225.424'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.769'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.417'MB/sec write SST data time = 0.02's, 4391.790'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 206966342.767'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64716 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.99 avg-zval = 1020.24 usrkeys = 64716 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.99 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.7500 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 9.932075 GB avg = 0.242 KB (by entry num) total key len = 1.062241 GB avg = 0.026 KB total ukey len = 0.734410 GB avg = 0.018 KB total ukey num = 0.040978868 Billion total entry num = 0.040978868 Billion write speed all = 42.375047844 MB/sec (with seq num) write speed all = 41.111499352 MB/sec (without seq num) 2024/11/19-15:22:56.424328 139850978166528 [compaction_job.cc:1948] [default] [JOB 176] Generated table #4982: 64716 keys, 66027146 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:56.424379 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000976424357, "cf_name": "default", "job": 176, "event": "table_file_creation", "file_number": 4982, "file_size": 66027146, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025856, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294320, "raw_average_key_size": 20, "raw_value_size": 65815297, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64716, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000976, "oldest_key_time": 0, "file_creation_time": 1732000976, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4982, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:56.438480 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004983.sst) /storage/mytopling/..//tmp/Topling-82038-oRMsYn 2024/11/19-15:22:56.466741 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-02 2024/11/19-15:22:56.466831 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00178/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:22:56.482705 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-01 2024/11/19-15:22:56.482785 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00179/att-01/compact.done) = No such file or directory: 11.203 sec on http probe failed: 5, giveup 2024/11/19-15:22:56.482809 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00179/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.810 sec, real 11.486 (submit 275.480 ms wait 11.203 s), input 126.810 MiB, send shutdown 2024/11/19-15:22:56.490136 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004983.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:22:56.491375 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-01 2024/11/19-15:22:56.491440 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00182/att-01/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:22:56.493264 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-01 2024/11/19-15:22:56.493456 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004983.sst): index pass time = 0.00's, 154.311'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63636 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:22:56.493529 139851158521600 [dcompact_etcd.cc:1773] job-00179/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:56.557007 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004983.sst): first pass time = 0.12's, 556.848'MB/sec 2024/11/19-15:22:56.557058 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004983.sst): old prealloc_size = 73819750, real_size = 65051408 2024/11/19-15:22:56.557111 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004983.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63636 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1017.03 avg-zval =1020.29 2024/11/19-15:22:56.557165 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004983.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.066 GB 2024/11/19-15:22:56.615394 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004983.sst): second pass time = 0.00's,231142421.429'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1115.149'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2154.574'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.383'MB/sec write SST data time = 0.02's, 3883.346'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 231142421.429'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63636 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 1020.29 usrkeys = 63636 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 1020.29 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7929 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0435 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 9.996795 GB avg = 0.244 KB (by entry num) total key len = 1.063514 GB avg = 0.026 KB total ukey len = 0.735174 GB avg = 0.018 KB total ukey num = 0.041042504 Billion total entry num = 0.041042504 Billion write speed all = 42.597447560 MB/sec (with seq num) write speed all = 41.332885488 MB/sec (without seq num) 2024/11/19-15:22:56.618820 139850978166528 [compaction_job.cc:1948] [default] [JOB 176] Generated table #4983: 63636 keys, 65036868 bytes (need compaction), temperature: kUnknown 2024/11/19-15:22:56.618869 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000976618847, "cf_name": "default", "job": 176, "event": "table_file_creation", "file_number": 4983, "file_size": 65036868, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64926912, "index_size": 108416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1272720, "raw_average_key_size": 20, "raw_value_size": 64719878, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63636, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000976, "oldest_key_time": 0, "file_creation_time": 1732000976, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4983, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:22:56.632814 139850978166528 [compaction_job.cc:703] job-00176: subcompact[0], size: 0.131064 G, files: /storage/mytopling/.rocksdb [004982,004983] 2024/11/19-15:22:56.633571 139850978166528 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16777215 2024/11/19-15:22:56.633701 139850978166528 (Original Log Time 2024/11/19-15:22:56.248476) EVENT_LOG_v1 {"time_micros": 1732000976248460, "job": 176, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4825, 4828, 4830, 4832], "score": 4.67102, "input_data_size": 134098880, "oldest_snapshot_seqno": -1} 2024/11/19-15:22:56.633707 139850978166528 (Original Log Time 2024/11/19-15:22:56.633234) [compaction_job.cc:2022] [default] [JOB 176] Compacted 4@1 files to L2 => 131064014 bytes 2024/11/19-15:22:56.633710 139850978166528 (Original Log Time 2024/11/19-15:22:56.633540) [compaction_job.cc:1224] [default] compacted to: files[5 71 67 34 0 0 0] max score 2.92 (119 files need compaction), MB/sec: 348.9 rd, 341.0 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128352, records dropped: 0 output_compression: Snappy 2024/11/19-15:22:56.633726 139850978166528 (Original Log Time 2024/11/19-15:22:56.633564) EVENT_LOG_v1 {"time_micros": 1732000976633552, "cf": "default", "job": 176, "event": "compaction_finished", "compaction_time_micros": 384352, "compaction_time_cpu_micros": 382442, "output_level": 2, "num_output_files": 2, "total_output_size": 131064014, "num_input_records": 128352, "num_output_records": 128352, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 71, 67, 34, 0, 0, 0]} 2024/11/19-15:22:56.638142 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004841.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:56.638166 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000976638163, "job": 176, "event": "table_file_deletion", "file_number": 4841} 2024/11/19-15:22:56.642410 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004846.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:56.642431 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000976642429, "job": 176, "event": "table_file_deletion", "file_number": 4846} 2024/11/19-15:22:56.646578 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004849.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:56.646598 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000976646595, "job": 176, "event": "table_file_deletion", "file_number": 4849} 2024/11/19-15:22:56.650845 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004853.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:22:56.650866 139850978166528 EVENT_LOG_v1 {"time_micros": 1732000976650864, "job": 176, "event": "table_file_deletion", "file_number": 4853} 2024/11/19-15:22:56.651026 139850978166528 [compaction_job.cc:2437] [default] [JOB 184] Compacting 4@1 files to L2, score 2.92, subcompactions 1 : 1 2024/11/19-15:22:56.651036 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 182 Base level 1, inputs: [4873(31MB) 4875(31MB) 4877(31MB) 4878(31MB)] 2024/11/19-15:22:56.778520 139851158521600 [dcompact_etcd.cc:1163] job-00179/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.285 sec, input = 126.810 MiB 2024/11/19-15:22:56.949446 139850978166528 [dcompact_etcd.cc:1163] job-00184/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.298 sec, input = 126.899 MiB 2024/11/19-15:22:57.835199 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-02 2024/11/19-15:22:57.835280 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00177/att-02/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:22:57.988964 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-01 2024/11/19-15:22:57.989049 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00180/att-01/compact.done) = No such file or directory: 11.197 sec on http probe failed: 5, giveup 2024/11/19-15:22:57.989066 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00180/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.844 sec, real 11.469 (submit 267.626 ms wait 11.197 s), input 126.844 MiB, send shutdown 2024/11/19-15:22:57.996495 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-01 2024/11/19-15:22:57.996715 139851148035840 [dcompact_etcd.cc:1773] job-00180/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:22:58.297518 139851148035840 [dcompact_etcd.cc:1163] job-00180/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.301 sec, input = 126.844 MiB 2024/11/19-15:22:58.369717 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-01 2024/11/19-15:22:58.369781 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00183/att-01/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:22:58.476261 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-02 2024/11/19-15:22:58.476367 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00178/att-02/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:22:58.499009 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-01 2024/11/19-15:22:58.499073 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00182/att-01/compact.done) = No such file or directory: 9.191 sec on http probe failed: 4, retry 2024/11/19-15:22:59.843857 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-02 2024/11/19-15:22:59.843939 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00177/att-02/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:22:59.957175 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-02 2024/11/19-15:22:59.957248 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00179/att-02/compact.done) = No such file or directory: 3.170 sec on http probe failed: 1, retry 2024/11/19-15:23:00.131366 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-00 2024/11/19-15:23:00.131454 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00184/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:23:00.378461 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-01 2024/11/19-15:23:00.378540 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00183/att-01/compact.done) = No such file or directory: 7.193 sec on http probe failed: 3, retry 2024/11/19-15:23:00.486751 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-02 2024/11/19-15:23:00.486838 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00178/att-02/compact.done) = No such file or directory: 7.193 sec on http probe failed: 3, retry 2024/11/19-15:23:00.505309 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-01 2024/11/19-15:23:00.505368 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00182/att-01/compact.done) = No such file or directory: 11.199 sec on http probe failed: 5, giveup 2024/11/19-15:23:00.505388 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00182/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.864 sec, real 11.490 (submit 285.452 ms wait 11.199 s), input 126.864 MiB, send shutdown 2024/11/19-15:23:00.512735 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-01 2024/11/19-15:23:00.512950 139851137550080 [dcompact_etcd.cc:1773] job-00182/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:00.784291 139851137550080 [dcompact_etcd.cc:1163] job-00182/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.271 sec, input = 126.864 MiB 2024/11/19-15:23:01.479067 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-02 2024/11/19-15:23:01.479194 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00180/att-02/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:23:01.544573 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #4984. Immutable memtables: 0. 2024/11/19-15:23:01.545018 139852362282752 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 16273898 2024/11/19-15:23:01.545105 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 185] Syncing log #4976 2024/11/19-15:23:01.545194 139850957195008 (Original Log Time 2024/11/19-15:23:01.545092) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:23:01.545202 139850957195008 [flush_job.cc:879] [default] [JOB 185] Flushing memtable with next log file: 4984 2024/11/19-15:23:01.545223 139850957195008 EVENT_LOG_v1 {"time_micros": 1732000981545217, "job": 185, "event": "flush_started", "num_memtables": 1, "num_entries": 1504564, "num_deletes": 0, "total_data_size": 456926230, "memory_usage": 538967432, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:23:01.545227 139850957195008 [flush_job.cc:910] [default] [JOB 185] Level-0 flush table #4985: started 2024/11/19-15:23:01.545578 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000023.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/004985.sst in ConvertToSST, mem_size = 513.999 M, time = 342.977 us 2024/11/19-15:23:01.851709 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-02 2024/11/19-15:23:01.851809 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00177/att-02/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:23:01.962541 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-02 2024/11/19-15:23:01.962725 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00179/att-02/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:23:02.065424 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/004985.sst): fsize = 514.001 M, time(ms): open: 0.035, seek: 0.014, finish: 0.037, meta: 0.013, Flush: 0.011, sync: 519.710, close: 0.010, all: 519.830 2024/11/19-15:23:02.066582 139850957195008 [flush_job.cc:1083] [default] [JOB 185] Flush lasted 521387 microseconds, and 198856 cpu microseconds. 2024/11/19-15:23:02.087363 139850957195008 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 15785681 2024/11/19-15:23:02.087443 139850957195008 (Original Log Time 2024/11/19-15:23:02.065468) [flush_job.cc:1029] [default] [JOB 185] Level-0 flush table #4985: 538968858 bytes OK (needs compaction) 2024/11/19-15:23:02.087449 139850957195008 (Original Log Time 2024/11/19-15:23:02.066599) [memtable_list.cc:560] [default] Level-0 commit table #4985 started 2024/11/19-15:23:02.087452 139850957195008 (Original Log Time 2024/11/19-15:23:02.087331) [memtable_list.cc:759] [default] Level-0 commit table #4985: memtable #1 done 2024/11/19-15:23:02.087455 139850957195008 (Original Log Time 2024/11/19-15:23:02.087357) EVENT_LOG_v1 {"time_micros": 1732000982087351, "job": 185, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [6, 71, 67, 34, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:23:02.087458 139850957195008 (Original Log Time 2024/11/19-15:23:02.087418) [db_impl_compaction_flush.cc:310] [default] Level summary: files[6 71 67 34 0 0 0] max score 2.67 (116 files need compaction) 2024/11/19-15:23:02.139884 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-00 2024/11/19-15:23:02.139983 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00184/att-00/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:23:02.387848 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-01 2024/11/19-15:23:02.387953 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00183/att-01/compact.done) = No such file or directory: 9.202 sec on http probe failed: 4, retry 2024/11/19-15:23:02.494765 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-02 2024/11/19-15:23:02.494861 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00178/att-02/compact.done) = No such file or directory: 9.203 sec on http probe failed: 4, retry 2024/11/19-15:23:03.487310 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-02 2024/11/19-15:23:03.487388 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00180/att-02/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:23:03.860238 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-02 2024/11/19-15:23:03.860331 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00177/att-02/compact.done) = No such file or directory: 11.206 sec on http probe failed: 5, giveup 2024/11/19-15:23:03.860350 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00177/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.970 sec, real 11.507 (submit 292.451 ms wait 11.206 s), input 126.970 MiB, send shutdown 2024/11/19-15:23:03.866976 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00177/att-02 2024/11/19-15:23:03.867206 139851169007360 [dcompact_etcd.cc:1773] job-00177/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:03.869307 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00177) = success 2024/11/19-15:23:03.869315 139851169007360 [dcompact_etcd.cc:1773] job-00177/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:23:03.869338 139851169007360 (Original Log Time 2024/11/19-15:22:29.291617) EVENT_LOG_v1 {"time_micros": 1732000949291611, "job": 177, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4835, 4837, 4839, 4842], "score": 4.42124, "input_data_size": 134097600, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:03.869345 139851169007360 [compaction_job.cc:2437] [default] [JOB 177] Compacting 4@1 files to L2, score 4.42, subcompactions 1 : 1 2024/11/19-15:23:03.869353 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 175 Base level 1, inputs: [4835(31MB) 4837(31MB) 4839(31MB) 4842(31MB)] 2024/11/19-15:23:03.869566 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004986.sst) /storage/mytopling/..//tmp/Topling-82038-8g55XR 2024/11/19-15:23:03.923122 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004986.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:23:03.923229 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004986.sst): index pass time = 0.00's,6515.334'MB/sec index type = UintIndex_AllOne usrkeys = 64716 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:23:03.965536 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-02 2024/11/19-15:23:03.965671 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00182/att-02/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:23:03.967105 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-02 2024/11/19-15:23:03.967171 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00179/att-02/compact.done) = No such file or directory: 7.184 sec on http probe failed: 3, retry 2024/11/19-15:23:03.989780 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004986.sst): first pass time = 0.12's, 558.302'MB/sec 2024/11/19-15:23:03.989839 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004986.sst): old prealloc_size = 73819750, real_size = 66042008 2024/11/19-15:23:03.989863 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004986.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64716 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.98 avg-zval =1020.23 2024/11/19-15:23:04.044754 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004986.sst): second pass time = 0.00's,146255446.667'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1202.657'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.713'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.409'MB/sec write SST data time = 0.02's, 4311.740'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 146255446.667'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64716 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.98 avg-zval = 1020.23 usrkeys = 64716 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.98 avg-zval = 1020.23 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.7500 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 10.062610 GB avg = 0.245 KB (by entry num) total key len = 1.064808 GB avg = 0.026 KB total ukey len = 0.735951 GB avg = 0.018 KB total ukey num = 0.041107220 Billion total entry num = 0.041107220 Billion write speed all = 41.663773596 MB/sec (with seq num) write speed all = 40.432449770 MB/sec (without seq num) 2024/11/19-15:23:04.048339 139851169007360 [compaction_job.cc:1948] [default] [JOB 177] Generated table #4986: 64716 keys, 66026826 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:04.048393 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000984048368, "cf_name": "default", "job": 177, "event": "table_file_creation", "file_number": 4986, "file_size": 66026826, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025504, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294320, "raw_average_key_size": 20, "raw_value_size": 65814951, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64716, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000983, "oldest_key_time": 0, "file_creation_time": 1732000983, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4986, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:04.062729 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004987.sst) /storage/mytopling/..//tmp/Topling-82038-tG7F25 2024/11/19-15:23:04.114031 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004987.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:23:04.119283 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004987.sst): index pass time = 0.01's, 97.682'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63675 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:23:04.148912 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-00 2024/11/19-15:23:04.149109 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00184/att-00/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:23:04.180221 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004987.sst): first pass time = 0.12's, 562.060'MB/sec 2024/11/19-15:23:04.180267 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004987.sst): old prealloc_size = 73819750, real_size = 65086800 2024/11/19-15:23:04.180323 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004987.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63675 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0648 GB zip-val = 0.0650 GB avg-val =1016.96 avg-zval =1020.22 2024/11/19-15:23:04.180367 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004987.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.066 GB 2024/11/19-15:23:04.238964 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004987.sst): second pass time = 0.00's,194459717.718'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1109.106'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2075.537'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.379'MB/sec write SST data time = 0.02's, 3918.676'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 194459717.718'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63675 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.96 avg-zval = 1020.22 usrkeys = 63675 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.96 avg-zval = 1020.22 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7933 UnZipSize{ index = 0.0008 GB value = 0.0648 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0650 GB dict = 0.00 MB all = 0.0651 GB } UnZip/Zip{ index = 7.0437 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 10.127365 GB avg = 0.246 KB (by entry num) total key len = 1.066082 GB avg = 0.026 KB total ukey len = 0.736715 GB avg = 0.018 KB total ukey num = 0.041170895 Billion total entry num = 0.041170895 Billion write speed all = 41.880547899 MB/sec (with seq num) write speed all = 40.648212831 MB/sec (without seq num) 2024/11/19-15:23:04.242445 139851169007360 [compaction_job.cc:1948] [default] [JOB 177] Generated table #4987: 63675 keys, 65072260 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:04.242501 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000984242477, "cf_name": "default", "job": 177, "event": "table_file_creation", "file_number": 4987, "file_size": 65072260, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64962240, "index_size": 108480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1273500, "raw_average_key_size": 20, "raw_value_size": 64755086, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63675, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000984, "oldest_key_time": 0, "file_creation_time": 1732000984, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4987, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:04.256508 139851169007360 [compaction_job.cc:703] job-00177: subcompact[0], size: 0.131099 G, files: /storage/mytopling/.rocksdb [004986,004987] 2024/11/19-15:23:04.257250 139851169007360 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16273897 2024/11/19-15:23:04.257408 139851169007360 (Original Log Time 2024/11/19-15:23:03.869383) EVENT_LOG_v1 {"time_micros": 1732000983869365, "job": 177, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4835, 4837, 4839, 4842], "score": 4.42124, "input_data_size": 134097600, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:04.257414 139851169007360 (Original Log Time 2024/11/19-15:23:04.256913) [compaction_job.cc:2022] [default] [JOB 177] Compacted 4@1 files to L2 => 131099086 bytes 2024/11/19-15:23:04.257418 139851169007360 (Original Log Time 2024/11/19-15:23:04.257220) [compaction_job.cc:1224] [default] compacted to: files[6 67 69 34 0 0 0] max score 2.67 (118 files need compaction), MB/sec: 346.4 rd, 338.6 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128391, records dropped: 0 output_compression: Snappy 2024/11/19-15:23:04.257436 139851169007360 (Original Log Time 2024/11/19-15:23:04.257243) EVENT_LOG_v1 {"time_micros": 1732000984257232, "cf": "default", "job": 177, "event": "compaction_finished", "compaction_time_micros": 387137, "compaction_time_cpu_micros": 385257, "output_level": 2, "num_output_files": 2, "total_output_size": 131099086, "num_input_records": 128391, "num_output_records": 128391, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 67, 69, 34, 0, 0, 0]} 2024/11/19-15:23:04.261706 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004857.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:04.261732 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000984261729, "job": 177, "event": "table_file_deletion", "file_number": 4857} 2024/11/19-15:23:04.265853 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004861.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:04.265874 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000984265871, "job": 177, "event": "table_file_deletion", "file_number": 4861} 2024/11/19-15:23:04.269949 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004865.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:04.269969 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000984269967, "job": 177, "event": "table_file_deletion", "file_number": 4865} 2024/11/19-15:23:04.274003 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004869.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:04.274023 139851169007360 EVENT_LOG_v1 {"time_micros": 1732000984274021, "job": 177, "event": "table_file_deletion", "file_number": 4869} 2024/11/19-15:23:04.274176 139851169007360 [compaction_job.cc:2437] [default] [JOB 186] Compacting 4@1 files to L2, score 2.67, subcompactions 1 : 1 2024/11/19-15:23:04.274185 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 184 Base level 1, inputs: [4879(31MB) 4881(31MB) 4883(31MB) 4884(31MB)] 2024/11/19-15:23:04.393494 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-01 2024/11/19-15:23:04.393579 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00183/att-01/compact.done) = No such file or directory: 11.211 sec on http probe failed: 5, giveup 2024/11/19-15:23:04.393597 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00183/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.966 sec, real 11.465 (submit 248.305 ms wait 11.211 s), input 126.966 MiB, send shutdown 2024/11/19-15:23:04.401720 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-01 2024/11/19-15:23:04.401938 139851179493120 [dcompact_etcd.cc:1773] job-00183/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:04.505149 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-02 2024/11/19-15:23:04.505206 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00178/att-02/compact.done) = No such file or directory: 11.211 sec on http probe failed: 5, giveup 2024/11/19-15:23:04.505221 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00178/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.907 sec, real 11.506 (submit 284.360 ms wait 11.211 s), input 126.907 MiB, send shutdown 2024/11/19-15:23:04.513771 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00178/att-02 2024/11/19-15:23:04.513973 139851127064320 [dcompact_etcd.cc:1773] job-00178/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:04.515796 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00178) = success 2024/11/19-15:23:04.515805 139851127064320 [dcompact_etcd.cc:1773] job-00178/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:23:04.515824 139851127064320 (Original Log Time 2024/11/19-15:22:30.036623) EVENT_LOG_v1 {"time_micros": 1732000950036616, "job": 178, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4845, 4847, 4848, 4850], "score": 4.17147, "input_data_size": 134098560, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:04.515832 139851127064320 [compaction_job.cc:2437] [default] [JOB 178] Compacting 4@1 files to L2, score 4.17, subcompactions 1 : 1 2024/11/19-15:23:04.515840 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 176 Base level 1, inputs: [4845(31MB) 4847(31MB) 4848(31MB) 4850(31MB)] 2024/11/19-15:23:04.516045 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004988.sst) /storage/mytopling/..//tmp/Topling-82038-uxCRDy 2024/11/19-15:23:04.541275 139851169007360 [dcompact_etcd.cc:1163] job-00186/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.267 sec, input = 126.950 MiB 2024/11/19-15:23:04.569304 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004988.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:23:04.572601 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004988.sst): index pass time = 0.00's, 157.947'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64715 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:23:04.635703 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004988.sst): first pass time = 0.12's, 560.900'MB/sec 2024/11/19-15:23:04.635754 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004988.sst): old prealloc_size = 73819750, real_size = 66151680 2024/11/19-15:23:04.635817 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004988.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64715 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.99 avg-zval =1020.24 2024/11/19-15:23:04.635863 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004988.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.067 GB 2024/11/19-15:23:04.692872 139851179493120 [dcompact_etcd.cc:1163] job-00183/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.291 sec, input = 126.966 MiB 2024/11/19-15:23:04.695729 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004988.sst): second pass time = 0.00's,232560399.293'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1102.968'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1851.737'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.382'MB/sec write SST data time = 0.02's, 3769.195'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 232560399.293'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64715 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 usrkeys = 64715 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0465 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 10.193180 GB avg = 0.247 KB (by entry num) total key len = 1.067376 GB avg = 0.026 KB total ukey len = 0.737491 GB avg = 0.018 KB total ukey num = 0.041235610 Billion total entry num = 0.041235610 Billion write speed all = 42.059755477 MB/sec (with seq num) write speed all = 40.827589172 MB/sec (without seq num) 2024/11/19-15:23:04.699229 139851127064320 [compaction_job.cc:1948] [default] [JOB 178] Generated table #4988: 64715 keys, 66136900 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:04.699280 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000984699256, "cf_name": "default", "job": 178, "event": "table_file_creation", "file_number": 4988, "file_size": 66136900, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025136, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294300, "raw_average_key_size": 20, "raw_value_size": 65814593, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64715, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000984, "oldest_key_time": 0, "file_creation_time": 1732000984, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4988, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:04.714536 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004989.sst) /storage/mytopling/..//tmp/Topling-82038-41ihoE 2024/11/19-15:23:04.765936 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004989.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:23:04.766032 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004989.sst): index pass time = 0.00's,7520.669'MB/sec index type = UintIndex_AllOne usrkeys = 63607 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:23:04.831277 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004989.sst): first pass time = 0.12's, 565.114'MB/sec 2024/11/19-15:23:04.831324 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004989.sst): old prealloc_size = 73819750, real_size = 64913992 2024/11/19-15:23:04.831346 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004989.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63607 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1017.04 avg-zval =1020.29 2024/11/19-15:23:04.885946 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004989.sst): second pass time = 0.00's,216357193.980'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1188.691'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.116'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.406'MB/sec write SST data time = 0.02's, 4213.754'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 216357193.980'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63607 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.04 avg-zval = 1020.29 usrkeys = 63607 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.04 avg-zval = 1020.29 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0000 GB value = 0.0649 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index =3975.4375 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 10.257870 GB avg = 0.248 KB (by entry num) total key len = 1.068648 GB avg = 0.026 KB total ukey len = 0.738255 GB avg = 0.018 KB total ukey num = 0.041299217 Billion total entry num = 0.041299217 Billion write speed all = 42.276101337 MB/sec (with seq num) write speed all = 41.042910503 MB/sec (without seq num) 2024/11/19-15:23:04.889549 139851127064320 [compaction_job.cc:1948] [default] [JOB 178] Generated table #4989: 63607 keys, 64899082 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:04.889600 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000984889577, "cf_name": "default", "job": 178, "event": "table_file_creation", "file_number": 4989, "file_size": 64899082, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64897744, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1272140, "raw_average_key_size": 20, "raw_value_size": 64690801, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63607, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000984, "oldest_key_time": 0, "file_creation_time": 1732000984, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4989, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:04.904850 139851127064320 [compaction_job.cc:703] job-00178: subcompact[0], size: 0.131036 G, files: /storage/mytopling/.rocksdb [004988,004989] 2024/11/19-15:23:04.905608 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777213 2024/11/19-15:23:04.905738 139851127064320 (Original Log Time 2024/11/19-15:23:04.515867) EVENT_LOG_v1 {"time_micros": 1732000984515851, "job": 178, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4845, 4847, 4848, 4850], "score": 4.17147, "input_data_size": 134098560, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:04.905744 139851127064320 (Original Log Time 2024/11/19-15:23:04.905284) [compaction_job.cc:2022] [default] [JOB 178] Compacted 4@1 files to L2 => 131035982 bytes 2024/11/19-15:23:04.905747 139851127064320 (Original Log Time 2024/11/19-15:23:04.905579) [compaction_job.cc:1224] [default] compacted to: files[6 63 71 34 0 0 0] max score 2.52 (116 files need compaction), MB/sec: 344.7 rd, 336.9 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128322, records dropped: 0 output_compression: Snappy 2024/11/19-15:23:04.905764 139851127064320 (Original Log Time 2024/11/19-15:23:04.905602) EVENT_LOG_v1 {"time_micros": 1732000984905590, "cf": "default", "job": 178, "event": "compaction_finished", "compaction_time_micros": 388996, "compaction_time_cpu_micros": 387407, "output_level": 2, "num_output_files": 2, "total_output_size": 131035982, "num_input_records": 128322, "num_output_records": 128322, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 63, 71, 34, 0, 0, 0]} 2024/11/19-15:23:04.909907 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004872.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:04.909942 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000984909939, "job": 178, "event": "table_file_deletion", "file_number": 4872} 2024/11/19-15:23:04.914098 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004874.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:04.914120 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000984914118, "job": 178, "event": "table_file_deletion", "file_number": 4874} 2024/11/19-15:23:04.918952 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004876.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:04.918971 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000984918969, "job": 178, "event": "table_file_deletion", "file_number": 4876} 2024/11/19-15:23:04.923931 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004880.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:04.923951 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000984923949, "job": 178, "event": "table_file_deletion", "file_number": 4880} 2024/11/19-15:23:04.924086 139851127064320 [compaction_job.cc:2437] [default] [JOB 187] Compacting 1@2 files to L3, score 2.52, subcompactions 1 : 1 2024/11/19-15:23:04.924094 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 185 Base level 2, inputs: [4937(3523B)] 2024/11/19-15:23:05.219665 139851127064320 [dcompact_etcd.cc:1163] job-00187/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.295 sec, input = 2.082 KiB 2024/11/19-15:23:05.229892 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-00 2024/11/19-15:23:05.229971 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00187/att-00/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:23:05.238209 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-00 2024/11/19-15:23:05.238260 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00187/att-00/compact.done) = No such file or directory: 0.010 sec on http probe failed: 2, retry 2024/11/19-15:23:05.243274 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-00 2024/11/19-15:23:05.243322 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00187/att-00/compact.done) = No such file or directory: 0.019 sec on http probe failed: 3, retry 2024/11/19-15:23:05.248019 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-00 2024/11/19-15:23:05.248065 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00187/att-00/compact.done) = No such file or directory: 0.024 sec on http probe failed: 4, retry 2024/11/19-15:23:05.255769 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-00 2024/11/19-15:23:05.255822 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00187/att-00/compact.done) = No such file or directory: 0.029 sec on http probe failed: 5, giveup 2024/11/19-15:23:05.255836 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00187/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.331 (submit 295.145 ms wait 0.029 s), input 2.082 KiB, send shutdown 2024/11/19-15:23:05.265084 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-00 2024/11/19-15:23:05.265296 139851127064320 [dcompact_etcd.cc:1773] job-00187/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:05.495491 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-02 2024/11/19-15:23:05.495571 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00180/att-02/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:23:05.545779 139851127064320 [dcompact_etcd.cc:1163] job-00187/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 2.082 KiB 2024/11/19-15:23:05.550825 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-01 2024/11/19-15:23:05.550882 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00187/att-01/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:23:05.557456 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-01 2024/11/19-15:23:05.557499 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00187/att-01/compact.done) = No such file or directory: 0.005 sec on http probe failed: 2, retry 2024/11/19-15:23:05.562064 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-01 2024/11/19-15:23:05.562105 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00187/att-01/compact.done) = No such file or directory: 0.012 sec on http probe failed: 3, retry 2024/11/19-15:23:05.570132 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-01 2024/11/19-15:23:05.570176 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00187/att-01/compact.done) = No such file or directory: 0.016 sec on http probe failed: 4, retry 2024/11/19-15:23:05.579667 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-01 2024/11/19-15:23:05.579709 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00187/att-01/compact.done) = No such file or directory: 0.025 sec on http probe failed: 5, giveup 2024/11/19-15:23:05.579722 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00187/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.314 (submit 280.390 ms wait 0.025 s), input 2.082 KiB, send shutdown 2024/11/19-15:23:05.587064 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-01 2024/11/19-15:23:05.587263 139851127064320 [dcompact_etcd.cc:1773] job-00187/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:05.863988 139851127064320 [dcompact_etcd.cc:1163] job-00187/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.277 sec, input = 2.082 KiB 2024/11/19-15:23:05.873822 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-02 2024/11/19-15:23:05.873877 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00187/att-02/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:23:05.881151 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-02 2024/11/19-15:23:05.881196 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00187/att-02/compact.done) = No such file or directory: 0.010 sec on http probe failed: 2, retry 2024/11/19-15:23:05.889812 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-02 2024/11/19-15:23:05.889856 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00187/att-02/compact.done) = No such file or directory: 0.017 sec on http probe failed: 3, retry 2024/11/19-15:23:05.898811 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-02 2024/11/19-15:23:05.898864 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00187/att-02/compact.done) = No such file or directory: 0.026 sec on http probe failed: 4, retry 2024/11/19-15:23:05.907260 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-02 2024/11/19-15:23:05.907313 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00187/att-02/compact.done) = No such file or directory: 0.035 sec on http probe failed: 5, giveup 2024/11/19-15:23:05.907328 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00187/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.320 (submit 276.643 ms wait 0.035 s), input 2.082 KiB, send shutdown 2024/11/19-15:23:05.915447 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00187/att-02 2024/11/19-15:23:05.915641 139851127064320 [dcompact_etcd.cc:1773] job-00187/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:05.917386 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00187) = success 2024/11/19-15:23:05.917394 139851127064320 [dcompact_etcd.cc:1773] job-00187/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:23:05.917411 139851127064320 (Original Log Time 2024/11/19-15:23:04.924107) EVENT_LOG_v1 {"time_micros": 1732000984924099, "job": 187, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4937], "score": 2.51841, "input_data_size": 3523, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:05.917418 139851127064320 [compaction_job.cc:2437] [default] [JOB 187] Compacting 1@2 files to L3, score 2.52, subcompactions 1 : 1 2024/11/19-15:23:05.917425 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 185 Base level 2, inputs: [4937(3523B)] 2024/11/19-15:23:05.917643 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004990.sst) /storage/mytopling/..//tmp/Topling-82038-O9qg0K 2024/11/19-15:23:05.917749 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004990.sst): first pass time = 0.00's, 21.845'MB/sec 2024/11/19-15:23:05.917768 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004990.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:23:05.917787 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004990.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, dictZip workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:23:05.917843 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004990.sst): index pass time = 0.00's, 1.101'MB/sec index type = UintIndex_AllOne usrkeys = 16 min-keylen = 4 max-keylen = 4 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 10.00 2024/11/19-15:23:05.938172 139851127064320 [top_zip_table_builder.cc:2752] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004990.sst): part-0: reject ZipStore ratio = 0.896 2024/11/19-15:23:05.938361 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004990.sst): old prealloc_size = 3875, real_size = 2528 2024/11/19-15:23:05.938378 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004990.sst): index type = UintIndex_AllOne , store type = ZipOffsetBlobStore usrkeys = 16 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 10.00 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 117.25 avg-zval = 135.00 2024/11/19-15:23:05.938513 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004990.sst): second pass time = 0.02's, 0.092'MB/sec, value only(88.0% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 24.345'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 11.918'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.526'MB/sec write SST data time = 0.00's, 28.991'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = 0.000'MB/sec zip my value time = 0.02's, unzip length = 0.000'GB zip my value throughput = 0.092'MB/sec zip pipeline throughput = 0.092'MB/sec entries = 16 avg-key = 16.00 avg-zkey = 12.00 avg-val = 117.25 avg-zval = 135.00 usrkeys = 16 avg-key = 8.00 avg-zkey = 12.00 avg-val = 117.25 avg-zval = 135.00 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 4 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.6667 value = 0.8685 dict = 0.00 all = 0.6178 } Zip/UnZip{ index = 1.5000 value = 1.1514 dict = 0.00 all = 1.6187 } ---------------------------- total value len = 10.257872 GB avg = 0.248 KB (by entry num) total key len = 1.068648 GB avg = 0.026 KB total ukey len = 0.738255 GB avg = 0.018 KB total ukey num = 0.041299233 Billion total entry num = 0.041299233 Billion write speed all = 42.110666430 MB/sec (with seq num) write speed all = 40.882301075 MB/sec (without seq num) 2024/11/19-15:23:05.938681 139851127064320 [compaction_job.cc:1948] [default] [JOB 187] Generated table #4990: 16 keys, 3451 bytes, temperature: kUnknown 2024/11/19-15:23:05.938715 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000985938695, "cf_name": "default", "job": 187, "event": "table_file_creation", "file_number": 4990, "file_size": 3451, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 2160, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 256, "raw_average_key_size": 16, "raw_value_size": 1876, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 16, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-ZipOffsetBlobStore", "creation_time": 1732000985, "oldest_key_time": 0, "file_creation_time": 1732000985, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4990, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:05.938810 139851127064320 [compaction_job.cc:703] job-00187: subcompact[0], size: 0.000003 G, files: /storage/mytopling/.rocksdb [004990] 2024/11/19-15:23:05.939273 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777216 2024/11/19-15:23:05.939369 139851127064320 (Original Log Time 2024/11/19-15:23:05.917450) EVENT_LOG_v1 {"time_micros": 1732000985917437, "job": 187, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4937], "score": 2.51841, "input_data_size": 3523, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:05.939375 139851127064320 (Original Log Time 2024/11/19-15:23:05.939005) [compaction_job.cc:2022] [default] [JOB 187] Compacted 1@2 files to L3 => 3451 bytes 2024/11/19-15:23:05.939378 139851127064320 (Original Log Time 2024/11/19-15:23:05.939250) [compaction_job.cc:1224] [default] compacted to: files[6 63 70 35 0 0 0] max score 2.52 (115 files need compaction), MB/sec: 0.2 rd, 0.2 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 16, records dropped: 0 output_compression: Snappy 2024/11/19-15:23:05.939389 139851127064320 (Original Log Time 2024/11/19-15:23:05.939268) EVENT_LOG_v1 {"time_micros": 1732000985939259, "cf": "default", "job": 187, "event": "compaction_finished", "compaction_time_micros": 21366, "compaction_time_cpu_micros": 1195, "output_level": 3, "num_output_files": 1, "total_output_size": 3451, "num_input_records": 16, "num_output_records": 16, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 63, 70, 35, 0, 0, 0]} 2024/11/19-15:23:05.939450 139851127064320 [compaction_job.cc:2437] [default] [JOB 188] Compacting 1@2 files to L3, score 2.52, subcompactions 1 : 1 2024/11/19-15:23:05.939457 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 186 Base level 2, inputs: [4938(31KB)] 2024/11/19-15:23:05.973235 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-02 2024/11/19-15:23:05.973286 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00179/att-02/compact.done) = No such file or directory: 9.189 sec on http probe failed: 4, retry 2024/11/19-15:23:05.973383 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-02 2024/11/19-15:23:05.973439 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00182/att-02/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:23:06.157146 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-00 2024/11/19-15:23:06.157214 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00184/att-00/compact.done) = No such file or directory: 9.200 sec on http probe failed: 4, retry 2024/11/19-15:23:06.252208 139851127064320 [dcompact_etcd.cc:1163] job-00188/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.312 sec, input = 23.008 KiB 2024/11/19-15:23:06.261644 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-00 2024/11/19-15:23:06.261700 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00188/att-00/compact.done) = No such file or directory: 0.001 sec on http probe failed: 1, retry 2024/11/19-15:23:06.271651 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-00 2024/11/19-15:23:06.271693 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00188/att-00/compact.done) = No such file or directory: 0.010 sec on http probe failed: 2, retry 2024/11/19-15:23:06.277562 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-00 2024/11/19-15:23:06.277606 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00188/att-00/compact.done) = No such file or directory: 0.020 sec on http probe failed: 3, retry 2024/11/19-15:23:06.282658 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-00 2024/11/19-15:23:06.282700 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00188/att-00/compact.done) = No such file or directory: 0.026 sec on http probe failed: 4, retry 2024/11/19-15:23:06.290253 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-00 2024/11/19-15:23:06.290295 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00188/att-00/compact.done) = No such file or directory: 0.031 sec on http probe failed: 5, giveup 2024/11/19-15:23:06.290308 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00188/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.350 (submit 312.339 ms wait 0.031 s), input 23.008 KiB, send shutdown 2024/11/19-15:23:06.298731 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-00 2024/11/19-15:23:06.299044 139851127064320 [dcompact_etcd.cc:1773] job-00188/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:06.606147 139851127064320 [dcompact_etcd.cc:1163] job-00188/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.307 sec, input = 23.008 KiB 2024/11/19-15:23:06.615486 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-01 2024/11/19-15:23:06.615546 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00188/att-01/compact.done) = No such file or directory: 0.001 sec on http probe failed: 1, retry 2024/11/19-15:23:06.620781 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-01 2024/11/19-15:23:06.620827 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00188/att-01/compact.done) = No such file or directory: 0.010 sec on http probe failed: 2, retry 2024/11/19-15:23:06.629755 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-01 2024/11/19-15:23:06.629800 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00188/att-01/compact.done) = No such file or directory: 0.015 sec on http probe failed: 3, retry 2024/11/19-15:23:06.637462 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-01 2024/11/19-15:23:06.637505 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00188/att-01/compact.done) = No such file or directory: 0.024 sec on http probe failed: 4, retry 2024/11/19-15:23:06.643157 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-01 2024/11/19-15:23:06.643202 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00188/att-01/compact.done) = No such file or directory: 0.032 sec on http probe failed: 5, giveup 2024/11/19-15:23:06.643216 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00188/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.344 (submit 307.012 ms wait 0.032 s), input 23.008 KiB, send shutdown 2024/11/19-15:23:06.651325 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-01 2024/11/19-15:23:06.742226 139851127064320 [dcompact_etcd.cc:1773] job-00188/att-01: CleanFiles 0.091 sec : 0.000 2024/11/19-15:23:07.041432 139851127064320 [dcompact_etcd.cc:1163] job-00188/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.299 sec, input = 23.008 KiB 2024/11/19-15:23:07.050764 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-02 2024/11/19-15:23:07.050820 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00188/att-02/compact.done) = No such file or directory: 0.001 sec on http probe failed: 1, retry 2024/11/19-15:23:07.055850 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-02 2024/11/19-15:23:07.055893 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00188/att-02/compact.done) = No such file or directory: 0.010 sec on http probe failed: 2, retry 2024/11/19-15:23:07.064996 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-02 2024/11/19-15:23:07.065041 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00188/att-02/compact.done) = No such file or directory: 0.015 sec on http probe failed: 3, retry 2024/11/19-15:23:07.073946 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-02 2024/11/19-15:23:07.073989 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00188/att-02/compact.done) = No such file or directory: 0.024 sec on http probe failed: 4, retry 2024/11/19-15:23:07.079659 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-02 2024/11/19-15:23:07.079705 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00188/att-02/compact.done) = No such file or directory: 0.033 sec on http probe failed: 5, giveup 2024/11/19-15:23:07.079719 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00188/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.337 (submit 299.086 ms wait 0.033 s), input 23.008 KiB, send shutdown 2024/11/19-15:23:07.087411 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00188/att-02 2024/11/19-15:23:07.131577 139851127064320 [dcompact_etcd.cc:1773] job-00188/att-02: CleanFiles 0.044 sec : 0.000 2024/11/19-15:23:07.179245 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00188) = success 2024/11/19-15:23:07.179257 139851127064320 [dcompact_etcd.cc:1773] job-00188/att-03: CleanFiles 0.048 sec : 0.000 2024/11/19-15:23:07.179275 139851127064320 (Original Log Time 2024/11/19-15:23:05.939467) EVENT_LOG_v1 {"time_micros": 1732000985939462, "job": 188, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4938], "score": 2.5184, "input_data_size": 32464, "oldest_snapshot_seqno": 138747049} 2024/11/19-15:23:07.179283 139851127064320 [compaction_job.cc:2437] [default] [JOB 188] Compacting 1@2 files to L3, score 2.52, subcompactions 1 : 1 2024/11/19-15:23:07.179291 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 186 Base level 2, inputs: [4938(31KB)] 2024/11/19-15:23:07.179537 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004991.sst) /storage/mytopling/..//tmp/Topling-82038-rGdDmf 2024/11/19-15:23:07.179712 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004991.sst): first pass time = 0.00's, 141.884'MB/sec 2024/11/19-15:23:07.179721 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004991.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:23:07.179753 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004991.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, dictZip workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:23:07.179902 139834035275520 [fixed_len_key_index.cc:363] FixedLenKeyIndex build_cache Finish: num_keys = 167, fixlen = 8, pref_len = 4, cplen = 3, time = 0.000 sec 2024/11/19-15:23:07.180021 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004991.sst): index pass time = 0.00's, 4.912'MB/sec index type = FixedLenKeyIndex usrkeys = 167 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 60.17 2024/11/19-15:23:07.200346 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004991.sst): old prealloc_size = 35710, real_size = 24416 2024/11/19-15:23:07.200368 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004991.sst): index type = FixedLenKeyIndex , store type = DictZipBlobStore usrkeys = 167 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 60.17 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 121.08 avg-zval = 84.79 2024/11/19-15:23:07.200534 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004991.sst): second pass time = 0.02's, 0.981'MB/sec, value only(85.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 146.961'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 563.987'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 5.990'MB/sec write SST data time = 0.00's, 170.034'MB/sec dict compress time = 0.00's, 3.079'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = 2.762'MB/sec zip my value time = 0.02's, unzip length = 0.000'GB zip my value throughput = 0.981'MB/sec zip pipeline throughput = 0.981'MB/sec entries = 167 avg-key = 20.00 avg-zkey = 60.17 avg-val = 121.08 avg-zval = 84.79 usrkeys = 167 avg-key = 12.00 avg-zkey = 60.17 avg-val = 121.08 avg-zval = 84.79 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 3 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.1994 value = 1.4280 dict = 1.00 all = 0.9146 } Zip/UnZip{ index = 5.0140 value = 0.7003 dict = 1.00 all = 1.0934 } ---------------------------- total value len = 10.257893 GB avg = 0.248 KB (by entry num) total key len = 1.068652 GB avg = 0.026 KB total ukey len = 0.738257 GB avg = 0.018 KB total ukey num = 0.041299400 Billion total entry num = 0.041299400 Billion write speed all = 41.914091270 MB/sec (with seq num) write speed all = 40.691457599 MB/sec (without seq num) 2024/11/19-15:23:07.200726 139851127064320 [compaction_job.cc:1948] [default] [JOB 188] Generated table #4991: 167 keys, 25761 bytes, temperature: kUnknown 2024/11/19-15:23:07.200767 139851127064320 EVENT_LOG_v1 {"time_micros": 1732000987200748, "cf_name": "default", "job": 188, "event": "table_file_creation", "file_number": 4991, "file_size": 25761, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 14160, "index_size": 10048, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3340, "raw_average_key_size": 20, "raw_value_size": 20220, "raw_average_value_size": 121, "num_data_blocks": 1, "num_entries": 167, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-FixedLenKeyIndex-DictZipBlobStore", "creation_time": 1732000987, "oldest_key_time": 0, "file_creation_time": 1732000987, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4991, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:07.200855 139851127064320 [compaction_job.cc:703] job-00188: subcompact[0], size: 0.000026 G, files: /storage/mytopling/.rocksdb [004991] 2024/11/19-15:23:07.201344 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777216 2024/11/19-15:23:07.201448 139851127064320 (Original Log Time 2024/11/19-15:23:07.179318) EVENT_LOG_v1 {"time_micros": 1732000987179303, "job": 188, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4938], "score": 2.5184, "input_data_size": 32464, "oldest_snapshot_seqno": 138747049} 2024/11/19-15:23:07.201453 139851127064320 (Original Log Time 2024/11/19-15:23:07.201059) [compaction_job.cc:2022] [default] [JOB 188] Compacted 1@2 files to L3 => 25761 bytes 2024/11/19-15:23:07.201456 139851127064320 (Original Log Time 2024/11/19-15:23:07.201322) [compaction_job.cc:1224] [default] compacted to: files[6 63 69 36 0 0 0] max score 2.52 (114 files need compaction), MB/sec: 1.5 rd, 1.2 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 167, records dropped: 0 output_compression: Snappy 2024/11/19-15:23:07.201469 139851127064320 (Original Log Time 2024/11/19-15:23:07.201339) EVENT_LOG_v1 {"time_micros": 1732000987201330, "cf": "default", "job": 188, "event": "compaction_finished", "compaction_time_micros": 21542, "compaction_time_cpu_micros": 1347, "output_level": 3, "num_output_files": 1, "total_output_size": 25761, "num_input_records": 167, "num_output_records": 167, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 63, 69, 36, 0, 0, 0]} 2024/11/19-15:23:07.201538 139851127064320 [compaction_job.cc:2437] [default] [JOB 189] Compacting 4@2 files to L3, score 2.52, subcompactions 1 : 1 2024/11/19-15:23:07.201545 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 187 Base level 2, inputs: [4953(62MB) 4954(61MB) 4956(62MB) 4959(61MB)] 2024/11/19-15:23:07.501529 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-02 2024/11/19-15:23:07.501600 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00180/att-02/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:23:07.521980 139851127064320 [dcompact_etcd.cc:1163] job-00189/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.320 sec, input = 254.478 MiB 2024/11/19-15:23:07.724435 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-00 2024/11/19-15:23:07.724541 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00186/att-00/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:23:07.875517 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-02 2024/11/19-15:23:07.875595 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00183/att-02/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:23:07.978101 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-02 2024/11/19-15:23:07.978158 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00179/att-02/compact.done) = No such file or directory: 11.195 sec on http probe failed: 5, giveup 2024/11/19-15:23:07.978174 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00179/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.810 sec, real 11.485 (submit 284.890 ms wait 11.195 s), input 126.810 MiB, send shutdown 2024/11/19-15:23:07.981002 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-02 2024/11/19-15:23:07.981062 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00182/att-02/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:23:07.982952 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00179/att-02 2024/11/19-15:23:07.983162 139851158521600 [dcompact_etcd.cc:1773] job-00179/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:07.985821 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00179) = success 2024/11/19-15:23:07.985829 139851158521600 [dcompact_etcd.cc:1773] job-00179/att-03: CleanFiles 0.003 sec : 0.000 2024/11/19-15:23:07.985849 139851158521600 (Original Log Time 2024/11/19-15:22:33.473125) EVENT_LOG_v1 {"time_micros": 1732000953473117, "job": 179, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4851, 4852, 4854, 4855], "score": 3.92169, "input_data_size": 134098816, "oldest_snapshot_seqno": 136998396} 2024/11/19-15:23:07.985857 139851158521600 [compaction_job.cc:2437] [default] [JOB 179] Compacting 4@1 files to L2, score 3.92, subcompactions 1 : 1 2024/11/19-15:23:07.985866 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 177 Base level 1, inputs: [4851(31MB) 4852(31MB) 4854(31MB) 4855(31MB)] 2024/11/19-15:23:07.986109 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004992.sst) /storage/mytopling/..//tmp/Topling-82038-R9jwV5 2024/11/19-15:23:08.040711 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004992.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:23:08.044037 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004992.sst): index pass time = 0.00's, 157.363'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64715 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:23:08.107770 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004992.sst): first pass time = 0.12's, 551.671'MB/sec 2024/11/19-15:23:08.107821 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004992.sst): old prealloc_size = 73819750, real_size = 66152528 2024/11/19-15:23:08.107879 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004992.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64715 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.00 avg-zval =1020.26 2024/11/19-15:23:08.107947 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004992.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.067 GB 2024/11/19-15:23:08.162031 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-00 2024/11/19-15:23:08.162104 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00184/att-00/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:23:08.162122 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00184/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.899 sec, real 11.511 (submit 297.983 ms wait 11.208 s), input 126.899 MiB, send shutdown 2024/11/19-15:23:08.167253 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-00 2024/11/19-15:23:08.167459 139850978166528 [dcompact_etcd.cc:1773] job-00184/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:08.168244 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004992.sst): second pass time = 0.00's,120541119.048'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1094.778'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2072.436'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.376'MB/sec write SST data time = 0.02's, 3816.262'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 120541119.048'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64715 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.26 usrkeys = 64715 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8063 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0465 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 10.323708 GB avg = 0.250 KB (by entry num) total key len = 1.069946 GB avg = 0.026 KB total ukey len = 0.739033 GB avg = 0.018 KB total ukey num = 0.041364115 Billion total entry num = 0.041364115 Billion write speed all = 42.011986941 MB/sec (with seq num) write speed all = 40.791806911 MB/sec (without seq num) 2024/11/19-15:23:08.171845 139851158521600 [compaction_job.cc:1948] [default] [JOB 179] Generated table #4992: 64715 keys, 66137732 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:08.171892 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000988171869, "cf_name": "default", "job": 179, "event": "table_file_creation", "file_number": 4992, "file_size": 66137732, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025984, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294300, "raw_average_key_size": 20, "raw_value_size": 65815451, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64715, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000987, "oldest_key_time": 0, "file_creation_time": 1732000987, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4992, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:08.186349 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004993.sst) /storage/mytopling/..//tmp/Topling-82038-BiZmr6 2024/11/19-15:23:08.238630 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004993.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:23:08.241886 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004993.sst): index pass time = 0.00's, 157.498'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63510 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:23:08.304581 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004993.sst): first pass time = 0.12's, 557.116'MB/sec 2024/11/19-15:23:08.304631 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004993.sst): old prealloc_size = 73819750, real_size = 64921024 2024/11/19-15:23:08.304689 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004993.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63510 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0646 GB zip-val = 0.0648 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:23:08.304753 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004993.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.066 GB 2024/11/19-15:23:08.363631 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004993.sst): second pass time = 0.00's,192806256.716'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1100.457'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2014.894'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.378'MB/sec write SST data time = 0.02's, 3844.691'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 192806256.716'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63510 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 63510 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7914 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0648 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index = 7.0421 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 10.388298 GB avg = 0.251 KB (by entry num) total key len = 1.071216 GB avg = 0.026 KB total ukey len = 0.739795 GB avg = 0.018 KB total ukey num = 0.041427625 Billion total entry num = 0.041427625 Billion write speed all = 42.224414971 MB/sec (with seq num) write speed all = 41.003241256 MB/sec (without seq num) 2024/11/19-15:23:08.367725 139851158521600 [compaction_job.cc:1948] [default] [JOB 179] Generated table #4993: 63510 keys, 64906500 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:08.367776 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000988367752, "cf_name": "default", "job": 179, "event": "table_file_creation", "file_number": 4993, "file_size": 64906500, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64796720, "index_size": 108224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1270200, "raw_average_key_size": 20, "raw_value_size": 64590096, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63510, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000988, "oldest_key_time": 0, "file_creation_time": 1732000988, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4993, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:08.381966 139851158521600 [compaction_job.cc:703] job-00179: subcompact[0], size: 0.131044 G, files: /storage/mytopling/.rocksdb [004992,004993] 2024/11/19-15:23:08.382692 139851158521600 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777216 2024/11/19-15:23:08.382837 139851158521600 (Original Log Time 2024/11/19-15:23:07.985897) EVENT_LOG_v1 {"time_micros": 1732000987985880, "job": 179, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4851, 4852, 4854, 4855], "score": 3.92169, "input_data_size": 134098816, "oldest_snapshot_seqno": 136998396} 2024/11/19-15:23:08.382844 139851158521600 (Original Log Time 2024/11/19-15:23:08.382377) [compaction_job.cc:2022] [default] [JOB 179] Compacted 4@1 files to L2 => 131044232 bytes 2024/11/19-15:23:08.382847 139851158521600 (Original Log Time 2024/11/19-15:23:08.382664) [compaction_job.cc:1224] [default] compacted to: files[6 59 71 36 0 0 0] max score 2.44 (112 files need compaction), MB/sec: 338.6 rd, 330.9 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128225, records dropped: 0 output_compression: Snappy 2024/11/19-15:23:08.382865 139851158521600 (Original Log Time 2024/11/19-15:23:08.382686) EVENT_LOG_v1 {"time_micros": 1732000988382675, "cf": "default", "job": 179, "event": "compaction_finished", "compaction_time_micros": 396081, "compaction_time_cpu_micros": 394102, "output_level": 2, "num_output_files": 2, "total_output_size": 131044232, "num_input_records": 128225, "num_output_records": 128225, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 59, 71, 36, 0, 0, 0]} 2024/11/19-15:23:08.386965 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004882.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:08.386987 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000988386984, "job": 179, "event": "table_file_deletion", "file_number": 4882} 2024/11/19-15:23:08.390842 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004886.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:08.390862 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000988390860, "job": 179, "event": "table_file_deletion", "file_number": 4886} 2024/11/19-15:23:08.394810 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004890.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:08.394831 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000988394829, "job": 179, "event": "table_file_deletion", "file_number": 4890} 2024/11/19-15:23:08.398766 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004893.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:08.398786 139851158521600 EVENT_LOG_v1 {"time_micros": 1732000988398784, "job": 179, "event": "table_file_deletion", "file_number": 4893} 2024/11/19-15:23:08.398930 139851158521600 [compaction_job.cc:2437] [default] [JOB 190] Compacting 2@2 files to L3, score 2.44, subcompactions 1 : 1 2024/11/19-15:23:08.398941 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 188 Base level 2, inputs: [4955(62MB) 4957(1217KB)] 2024/11/19-15:23:08.443984 139850978166528 [dcompact_etcd.cc:1163] job-00184/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.276 sec, input = 126.899 MiB 2024/11/19-15:23:08.712371 139851158521600 [dcompact_etcd.cc:1163] job-00190/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.313 sec, input = 65.224 MiB 2024/11/19-15:23:09.509494 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-02 2024/11/19-15:23:09.509571 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00180/att-02/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:23:09.509588 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00180/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.844 sec, real 11.513 (submit 300.724 ms wait 11.204 s), input 126.844 MiB, send shutdown 2024/11/19-15:23:09.514406 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00180/att-02 2024/11/19-15:23:09.514621 139851148035840 [dcompact_etcd.cc:1773] job-00180/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:09.516533 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00180) = success 2024/11/19-15:23:09.516541 139851148035840 [dcompact_etcd.cc:1773] job-00180/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:23:09.516560 139851148035840 (Original Log Time 2024/11/19-15:22:34.998779) EVENT_LOG_v1 {"time_micros": 1732000954998772, "job": 180, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4856, 4858, 4859, 4860], "score": 3.67191, "input_data_size": 134099840, "oldest_snapshot_seqno": 137127980} 2024/11/19-15:23:09.516568 139851148035840 [compaction_job.cc:2437] [default] [JOB 180] Compacting 4@1 files to L2, score 3.67, subcompactions 1 : 1 2024/11/19-15:23:09.516576 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 178 Base level 1, inputs: [4856(31MB) 4858(31MB) 4859(31MB) 4860(31MB)] 2024/11/19-15:23:09.516820 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004994.sst) /storage/mytopling/..//tmp/Topling-82038-MMKHTV 2024/11/19-15:23:09.569980 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004994.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:23:09.570093 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004994.sst): index pass time = 0.00's,6070.327'MB/sec index type = UintIndex_AllOne usrkeys = 64715 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:23:09.637326 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004994.sst): first pass time = 0.12's, 556.954'MB/sec 2024/11/19-15:23:09.637378 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004994.sst): old prealloc_size = 73819750, real_size = 66041912 2024/11/19-15:23:09.637404 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004994.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64715 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.00 avg-zval =1020.25 2024/11/19-15:23:09.691387 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004994.sst): second pass time = 0.00's,140031629.787'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1222.766'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.890'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.415'MB/sec write SST data time = 0.02's, 4399.136'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 140031629.787'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64715 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.00 avg-zval = 1020.25 usrkeys = 64715 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.00 avg-zval = 1020.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.6875 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 10.454113 GB avg = 0.252 KB (by entry num) total key len = 1.072511 GB avg = 0.026 KB total ukey len = 0.740572 GB avg = 0.018 KB total ukey num = 0.041492340 Billion total entry num = 0.041492340 Billion write speed all = 42.264915033 MB/sec (with seq num) write speed all = 41.047788295 MB/sec (without seq num) 2024/11/19-15:23:09.695304 139851148035840 [compaction_job.cc:1948] [default] [JOB 180] Generated table #4994: 64715 keys, 66026698 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:09.695358 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000989695331, "cf_name": "default", "job": 180, "event": "table_file_creation", "file_number": 4994, "file_size": 66026698, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025408, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294300, "raw_average_key_size": 20, "raw_value_size": 65814866, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64715, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000989, "oldest_key_time": 0, "file_creation_time": 1732000989, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4994, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:09.710722 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004995.sst) /storage/mytopling/..//tmp/Topling-82038-Cex4Hh 2024/11/19-15:23:09.729364 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-00 2024/11/19-15:23:09.729436 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00186/att-00/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:23:09.762970 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004995.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:23:09.766673 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004995.sst): index pass time = 0.00's, 138.216'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63547 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:23:09.828500 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004995.sst): first pass time = 0.12's, 559.576'MB/sec 2024/11/19-15:23:09.828548 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004995.sst): old prealloc_size = 73819750, real_size = 64956320 2024/11/19-15:23:09.828605 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004995.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63547 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0646 GB zip-val = 0.0648 GB avg-val =1016.97 avg-zval =1020.22 2024/11/19-15:23:09.828662 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004995.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.066 GB 2024/11/19-15:23:09.883489 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-02 2024/11/19-15:23:09.883565 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00183/att-02/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:23:09.892025 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004995.sst): second pass time = 0.00's,128224621.032'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1023.342'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2052.814'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.337'MB/sec write SST data time = 0.02's, 3990.152'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 128224621.032'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63547 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.97 avg-zval = 1020.22 usrkeys = 63547 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.97 avg-zval = 1020.22 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7919 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0648 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index = 7.0420 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 10.518738 GB avg = 0.253 KB (by entry num) total key len = 1.073782 GB avg = 0.026 KB total ukey len = 0.741334 GB avg = 0.018 KB total ukey num = 0.041555887 Billion total entry num = 0.041555887 Billion write speed all = 42.475289622 MB/sec (with seq num) write speed all = 41.257194939 MB/sec (without seq num) 2024/11/19-15:23:09.896973 139851148035840 [compaction_job.cc:1948] [default] [JOB 180] Generated table #4995: 63547 keys, 64941828 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:09.897022 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000989896998, "cf_name": "default", "job": 180, "event": "table_file_creation", "file_number": 4995, "file_size": 64941828, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64831952, "index_size": 108288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1270940, "raw_average_key_size": 20, "raw_value_size": 64625209, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63547, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000989, "oldest_key_time": 0, "file_creation_time": 1732000989, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4995, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:09.911791 139851148035840 [compaction_job.cc:703] job-00180: subcompact[0], size: 0.130969 G, files: /storage/mytopling/.rocksdb [004994,004995] 2024/11/19-15:23:09.912486 139851148035840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777216 2024/11/19-15:23:09.912632 139851148035840 (Original Log Time 2024/11/19-15:23:09.516603) EVENT_LOG_v1 {"time_micros": 1732000989516587, "job": 180, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4856, 4858, 4859, 4860], "score": 3.67191, "input_data_size": 134099840, "oldest_snapshot_seqno": 137127980} 2024/11/19-15:23:09.912642 139851148035840 (Original Log Time 2024/11/19-15:23:09.912179) [compaction_job.cc:2022] [default] [JOB 180] Compacted 4@1 files to L2 => 130968526 bytes 2024/11/19-15:23:09.912646 139851148035840 (Original Log Time 2024/11/19-15:23:09.912455) [compaction_job.cc:1224] [default] compacted to: files[6 55 73 36 0 0 0] max score 2.48 (112 files need compaction), MB/sec: 339.3 rd, 331.4 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(124.9 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128262, records dropped: 0 output_compression: Snappy 2024/11/19-15:23:09.912672 139851148035840 (Original Log Time 2024/11/19-15:23:09.912479) EVENT_LOG_v1 {"time_micros": 1732000989912466, "cf": "default", "job": 180, "event": "compaction_finished", "compaction_time_micros": 395199, "compaction_time_cpu_micros": 392587, "output_level": 2, "num_output_files": 2, "total_output_size": 130968526, "num_input_records": 128262, "num_output_records": 128262, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 55, 73, 36, 0, 0, 0]} 2024/11/19-15:23:09.916762 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004896.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:09.916785 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000989916783, "job": 180, "event": "table_file_deletion", "file_number": 4896} 2024/11/19-15:23:09.922414 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004899.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:09.922435 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000989922433, "job": 180, "event": "table_file_deletion", "file_number": 4899} 2024/11/19-15:23:09.926447 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004905.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:09.926466 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000989926464, "job": 180, "event": "table_file_deletion", "file_number": 4905} 2024/11/19-15:23:09.930464 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004908.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:09.930483 139851148035840 EVENT_LOG_v1 {"time_micros": 1732000989930481, "job": 180, "event": "table_file_deletion", "file_number": 4908} 2024/11/19-15:23:09.930616 139851148035840 [compaction_job.cc:2437] [default] [JOB 191] Compacting 2@2 files to L3, score 2.48, subcompactions 1 : 1 2024/11/19-15:23:09.930624 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 189 Base level 2, inputs: [4958(12MB) 4962(16MB)] 2024/11/19-15:23:09.985725 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-02 2024/11/19-15:23:09.985781 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00182/att-02/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:23:10.234468 139851148035840 [dcompact_etcd.cc:1163] job-00191/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.303 sec, input = 35.565 MiB 2024/11/19-15:23:10.352808 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-00 2024/11/19-15:23:10.352883 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00190/att-00/compact.done) = No such file or directory: 1.631 sec on http probe failed: 1, retry 2024/11/19-15:23:11.132027 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-00 2024/11/19-15:23:11.132117 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00191/att-00/compact.done) = No such file or directory: 0.889 sec on http probe failed: 1, retry 2024/11/19-15:23:11.625442 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-01 2024/11/19-15:23:11.625529 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00184/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:23:11.664860 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-00 2024/11/19-15:23:11.664928 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00190/att-00/compact.done) = No such file or directory: 2.945 sec on http probe failed: 2, retry 2024/11/19-15:23:11.735325 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-00 2024/11/19-15:23:11.735392 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00186/att-00/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:23:11.848894 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-00 2024/11/19-15:23:11.848978 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00191/att-00/compact.done) = No such file or directory: 1.609 sec on http probe failed: 2, retry 2024/11/19-15:23:11.888375 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-02 2024/11/19-15:23:11.888434 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00183/att-02/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:23:11.990478 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-02 2024/11/19-15:23:11.990536 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00182/att-02/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:23:11.990553 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00182/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.864 sec, real 11.478 (submit 271.251 ms wait 11.202 s), input 126.864 MiB, send shutdown 2024/11/19-15:23:11.994939 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00182/att-02 2024/11/19-15:23:11.995149 139851137550080 [dcompact_etcd.cc:1773] job-00182/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:11.995384 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00182) = success 2024/11/19-15:23:11.995390 139851137550080 [dcompact_etcd.cc:1773] job-00182/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:11.995410 139851137550080 (Original Log Time 2024/11/19-15:22:37.481928) EVENT_LOG_v1 {"time_micros": 1732000957481913, "job": 182, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4862, 4863, 4864, 4866], "score": 3.42213, "input_data_size": 134099264, "oldest_snapshot_seqno": 137364401} 2024/11/19-15:23:11.995417 139851137550080 [compaction_job.cc:2437] [default] [JOB 182] Compacting 4@1 files to L2, score 3.42, subcompactions 1 : 1 2024/11/19-15:23:11.995425 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 180 Base level 1, inputs: [4862(31MB) 4863(31MB) 4864(31MB) 4866(31MB)] 2024/11/19-15:23:11.995676 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004996.sst) /storage/mytopling/..//tmp/Topling-82038-r8eACc 2024/11/19-15:23:12.050435 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004996.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:23:12.055833 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004996.sst): index pass time = 0.01's, 96.596'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64715 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:23:12.118744 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004996.sst): first pass time = 0.12's, 545.360'MB/sec 2024/11/19-15:23:12.118794 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004996.sst): old prealloc_size = 73819750, real_size = 66151760 2024/11/19-15:23:12.118850 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004996.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64715 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.99 avg-zval =1020.25 2024/11/19-15:23:12.118910 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004996.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.068 sec, Key+Value = 0.067 GB 2024/11/19-15:23:12.179316 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004996.sst): second pass time = 0.00's,146254846.667'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1093.083'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2138.466'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.375'MB/sec write SST data time = 0.02's, 3820.470'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 146254846.667'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64715 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.25 usrkeys = 64715 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8063 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0465 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 10.584553 GB avg = 0.254 KB (by entry num) total key len = 1.075076 GB avg = 0.026 KB total ukey len = 0.742111 GB avg = 0.018 KB total ukey num = 0.041620602 Billion total entry num = 0.041620602 Billion write speed all = 42.366121223 MB/sec (with seq num) write speed all = 41.156269000 MB/sec (without seq num) 2024/11/19-15:23:12.269735 139851137550080 [compaction_job.cc:1948] [default] [JOB 182] Generated table #4996: 64715 keys, 66136964 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:12.269793 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000992269766, "cf_name": "default", "job": 182, "event": "table_file_creation", "file_number": 4996, "file_size": 66136964, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025216, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294300, "raw_average_key_size": 20, "raw_value_size": 65814681, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64715, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000991, "oldest_key_time": 0, "file_creation_time": 1732000991, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4996, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:12.284519 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004997.sst) /storage/mytopling/..//tmp/Topling-82038-KlwNLu 2024/11/19-15:23:12.336579 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004997.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:23:12.336684 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004997.sst): index pass time = 0.00's,7234.610'MB/sec index type = UintIndex_AllOne usrkeys = 63566 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:23:12.402487 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004997.sst): first pass time = 0.12's, 558.871'MB/sec 2024/11/19-15:23:12.402542 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004997.sst): old prealloc_size = 73819750, real_size = 64869816 2024/11/19-15:23:12.402564 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004997.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63566 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0646 GB zip-val = 0.0649 GB avg-val =1017.00 avg-zval =1020.26 2024/11/19-15:23:12.456225 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004997.sst): second pass time = 0.00's,165337000.000'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1208.463'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.955'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.416'MB/sec write SST data time = 0.02's, 4207.051'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 165337000.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63566 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.00 avg-zval = 1020.26 usrkeys = 63566 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.00 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0000 GB value = 0.0649 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index =3972.8750 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 10.649200 GB avg = 0.255 KB (by entry num) total key len = 1.076347 GB avg = 0.026 KB total ukey len = 0.742874 GB avg = 0.018 KB total ukey num = 0.041684168 Billion total entry num = 0.041684168 Billion write speed all = 42.562814234 MB/sec (with seq num) write speed all = 41.352332182 MB/sec (without seq num) 2024/11/19-15:23:12.473648 139851137550080 [compaction_job.cc:1948] [default] [JOB 182] Generated table #4997: 63566 keys, 64854858 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:12.473705 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000992473682, "cf_name": "default", "job": 182, "event": "table_file_creation", "file_number": 4997, "file_size": 64854858, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64853568, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1271320, "raw_average_key_size": 20, "raw_value_size": 64646767, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63566, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000992, "oldest_key_time": 0, "file_creation_time": 1732000992, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4997, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:12.488133 139851137550080 [compaction_job.cc:703] job-00182: subcompact[0], size: 0.130992 G, files: /storage/mytopling/.rocksdb [004996,004997] 2024/11/19-15:23:12.489534 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777216 2024/11/19-15:23:12.489794 139851137550080 (Original Log Time 2024/11/19-15:23:11.995453) EVENT_LOG_v1 {"time_micros": 1732000991995437, "job": 182, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4862, 4863, 4864, 4866], "score": 3.42213, "input_data_size": 134099264, "oldest_snapshot_seqno": 137364401} 2024/11/19-15:23:12.489801 139851137550080 (Original Log Time 2024/11/19-15:23:12.489209) [compaction_job.cc:2022] [default] [JOB 182] Compacted 4@1 files to L2 => 130991822 bytes 2024/11/19-15:23:12.489804 139851137550080 (Original Log Time 2024/11/19-15:23:12.489501) [compaction_job.cc:1224] [default] compacted to: files[6 51 75 36 0 0 0] max score 2.53 (112 files need compaction), MB/sec: 272.2 rd, 265.9 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(124.9 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128281, records dropped: 0 output_compression: Snappy 2024/11/19-15:23:12.489827 139851137550080 (Original Log Time 2024/11/19-15:23:12.489527) EVENT_LOG_v1 {"time_micros": 1732000992489514, "cf": "default", "job": 182, "event": "compaction_finished", "compaction_time_micros": 492697, "compaction_time_cpu_micros": 390645, "output_level": 2, "num_output_files": 2, "total_output_size": 130991822, "num_input_records": 128281, "num_output_records": 128281, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 51, 75, 36, 0, 0, 0]} 2024/11/19-15:23:12.531230 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-00 2024/11/19-15:23:12.531321 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00189/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:23:12.561079 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004818.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:12.561104 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000992561101, "job": 182, "event": "table_file_deletion", "file_number": 4818} 2024/11/19-15:23:12.565560 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004821.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:12.565583 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000992565581, "job": 182, "event": "table_file_deletion", "file_number": 4821} 2024/11/19-15:23:12.569052 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-00 2024/11/19-15:23:12.569119 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00191/att-00/compact.done) = No such file or directory: 2.326 sec on http probe failed: 3, retry 2024/11/19-15:23:12.570200 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004823.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:12.570221 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000992570219, "job": 182, "event": "table_file_deletion", "file_number": 4823} 2024/11/19-15:23:12.570681 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004920.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:12.570694 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000992570692, "job": 182, "event": "table_file_deletion", "file_number": 4920} 2024/11/19-15:23:12.570848 139851137550080 [compaction_job.cc:2437] [default] [JOB 192] Compacting 1@2 files to L3, score 2.53, subcompactions 1 : 1 2024/11/19-15:23:12.570856 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 190 Base level 2, inputs: [4963(669KB)] 2024/11/19-15:23:12.916941 139851137550080 [dcompact_etcd.cc:1163] job-00192/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.346 sec, input = 3.419 MiB 2024/11/19-15:23:12.975453 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-00 2024/11/19-15:23:12.975519 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00190/att-00/compact.done) = No such file or directory: 4.257 sec on http probe failed: 3, retry 2024/11/19-15:23:13.011500 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-00 2024/11/19-15:23:13.011562 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00192/att-00/compact.done) = No such file or directory: 0.086 sec on http probe failed: 1, retry 2024/11/19-15:23:13.088052 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-00 2024/11/19-15:23:13.088116 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00192/att-00/compact.done) = No such file or directory: 0.163 sec on http probe failed: 2, retry 2024/11/19-15:23:13.164447 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-00 2024/11/19-15:23:13.164499 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00192/att-00/compact.done) = No such file or directory: 0.240 sec on http probe failed: 3, retry 2024/11/19-15:23:13.242536 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-00 2024/11/19-15:23:13.242588 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00192/att-00/compact.done) = No such file or directory: 0.316 sec on http probe failed: 4, retry 2024/11/19-15:23:13.288613 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-00 2024/11/19-15:23:13.288680 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00191/att-00/compact.done) = No such file or directory: 3.046 sec on http probe failed: 4, retry 2024/11/19-15:23:13.320161 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-00 2024/11/19-15:23:13.320209 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00192/att-00/compact.done) = No such file or directory: 0.394 sec on http probe failed: 5, giveup 2024/11/19-15:23:13.320223 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00192/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.749 (submit 345.641 ms wait 0.394 s), input 3.419 MiB, send shutdown 2024/11/19-15:23:13.328208 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-00 2024/11/19-15:23:13.328411 139851137550080 [dcompact_etcd.cc:1773] job-00192/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:13.630976 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-01 2024/11/19-15:23:13.631099 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00184/att-01/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:23:13.740951 139851137550080 [dcompact_etcd.cc:1163] job-00192/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.412 sec, input = 3.419 MiB 2024/11/19-15:23:13.744115 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-00 2024/11/19-15:23:13.744193 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00186/att-00/compact.done) = No such file or directory: 9.194 sec on http probe failed: 4, retry 2024/11/19-15:23:13.835121 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-01 2024/11/19-15:23:13.835199 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00192/att-01/compact.done) = No such file or directory: 0.086 sec on http probe failed: 1, retry 2024/11/19-15:23:13.897107 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-02 2024/11/19-15:23:13.897175 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00183/att-02/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:23:13.912928 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-01 2024/11/19-15:23:13.912991 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00192/att-01/compact.done) = No such file or directory: 0.163 sec on http probe failed: 2, retry 2024/11/19-15:23:13.989789 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-01 2024/11/19-15:23:13.989868 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00192/att-01/compact.done) = No such file or directory: 0.241 sec on http probe failed: 3, retry 2024/11/19-15:23:14.005558 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-00 2024/11/19-15:23:14.005634 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00191/att-00/compact.done) = No such file or directory: 3.766 sec on http probe failed: 5, giveup 2024/11/19-15:23:14.005652 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00191/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 35.565 sec, real 4.075 (submit 303.340 ms wait 3.766 s), input 35.565 MiB, send shutdown 2024/11/19-15:23:14.010683 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-00 2024/11/19-15:23:14.010900 139851148035840 [dcompact_etcd.cc:1773] job-00191/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:14.063486 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-01 2024/11/19-15:23:14.063575 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00192/att-01/compact.done) = No such file or directory: 0.317 sec on http probe failed: 4, retry 2024/11/19-15:23:14.140557 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-01 2024/11/19-15:23:14.140629 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00192/att-01/compact.done) = No such file or directory: 0.391 sec on http probe failed: 5, giveup 2024/11/19-15:23:14.140651 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00192/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.812 (submit 412.462 ms wait 0.391 s), input 3.419 MiB, send shutdown 2024/11/19-15:23:14.149692 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-01 2024/11/19-15:23:14.150006 139851137550080 [dcompact_etcd.cc:1773] job-00192/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:14.294872 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-00 2024/11/19-15:23:14.294950 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00190/att-00/compact.done) = No such file or directory: 5.568 sec on http probe failed: 4, retry 2024/11/19-15:23:14.299906 139851148035840 [dcompact_etcd.cc:1163] job-00191/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 35.565 MiB 2024/11/19-15:23:14.425813 139851137550080 [dcompact_etcd.cc:1163] job-00192/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.276 sec, input = 3.419 MiB 2024/11/19-15:23:14.516407 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-02 2024/11/19-15:23:14.516490 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00192/att-02/compact.done) = No such file or directory: 0.086 sec on http probe failed: 1, retry 2024/11/19-15:23:14.536801 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-00 2024/11/19-15:23:14.536851 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00189/att-00/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:23:14.589599 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-02 2024/11/19-15:23:14.589652 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00192/att-02/compact.done) = No such file or directory: 0.159 sec on http probe failed: 2, retry 2024/11/19-15:23:14.662674 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-02 2024/11/19-15:23:14.662741 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00192/att-02/compact.done) = No such file or directory: 0.232 sec on http probe failed: 3, retry 2024/11/19-15:23:14.736330 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-02 2024/11/19-15:23:14.736391 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00192/att-02/compact.done) = No such file or directory: 0.305 sec on http probe failed: 4, retry 2024/11/19-15:23:14.810427 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-02 2024/11/19-15:23:14.810501 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00192/att-02/compact.done) = No such file or directory: 0.379 sec on http probe failed: 5, giveup 2024/11/19-15:23:14.810518 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00192/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.660 (submit 275.718 ms wait 0.379 s), input 3.419 MiB, send shutdown 2024/11/19-15:23:14.818522 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00192/att-02 2024/11/19-15:23:14.818743 139851137550080 [dcompact_etcd.cc:1773] job-00192/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:14.820498 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00192) = success 2024/11/19-15:23:14.820508 139851137550080 [dcompact_etcd.cc:1773] job-00192/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:23:14.820528 139851137550080 (Original Log Time 2024/11/19-15:23:12.570869) EVENT_LOG_v1 {"time_micros": 1732000992570862, "job": 192, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4963], "score": 2.53482, "input_data_size": 685094, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:14.820535 139851137550080 [compaction_job.cc:2437] [default] [JOB 192] Compacting 1@2 files to L3, score 2.53, subcompactions 1 : 1 2024/11/19-15:23:14.820543 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 190 Base level 2, inputs: [4963(669KB)] 2024/11/19-15:23:14.820812 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004998.sst) /storage/mytopling/..//tmp/Topling-82038-MD7CqC 2024/11/19-15:23:14.860481 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004998.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0013 GB, waited 0.000 sec, Key+Value = 0.004 GB 2024/11/19-15:23:14.863487 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004998.sst): first pass time = 0.04's, 84.044'MB/sec 2024/11/19-15:23:14.869368 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004998.sst): index pass time = 0.01's, 202.477'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 149400 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0018 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:23:14.869441 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004998.sst): old prealloc_size = 753603, real_size = 290976 2024/11/19-15:23:14.869520 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004998.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZeroLengthBlobStore usrkeys = 149400 prefix = 4 raw-key = 0.0018 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 0.00 avg-zval = 0.00 2024/11/19-15:23:14.869573 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004998.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.009 sec, Key+Value = 0.004 GB 2024/11/19-15:23:14.882092 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004998.sst): second pass time = 0.00's, 0.000'MB/sec, value only( 0.0% of KV) wait indexing time = 0.01's, remap KeyValue time = 0.01's, 20.199'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3438.124'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 153.133'MB/sec write SST data time = 0.01's, 20.717'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 0.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 149400 avg-key = 24.00 avg-zkey = 1.70 avg-val = 0.00 avg-zval = 0.00 usrkeys = 149400 avg-key = 16.00 avg-zkey = 1.70 avg-val = 0.00 avg-zval = 0.00 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 5 UnZipSize{ index = 0.0024 GB value = 0.0000 GB dict = 0.00 MB all = 0.0036 GB } __ZipSize{ index = 0.0003 GB value = 0.0000 GB dict = 0.00 MB all = 0.0003 GB } UnZip/Zip{ index = 9.4342 value = 0.0000 dict = 0.00 all = 14.0329 } Zip/UnZip{ index = 0.1060 value = inf dict = 0.00 all = 0.0713 } ---------------------------- total value len = 10.649200 GB avg = 0.255 KB (by entry num) total key len = 1.079933 GB avg = 0.026 KB total ukey len = 0.745264 GB avg = 0.018 KB total ukey num = 0.041833568 Billion total entry num = 0.041833568 Billion write speed all = 42.204190577 MB/sec (with seq num) write speed all = 40.999974060 MB/sec (without seq num) 2024/11/19-15:23:14.882413 139851137550080 [compaction_job.cc:1948] [default] [JOB 192] Generated table #4998: 149400 keys, 255513 bytes, temperature: kUnknown 2024/11/19-15:23:14.882458 139851137550080 EVENT_LOG_v1 {"time_micros": 1732000994882437, "cf_name": "default", "job": 192, "event": "table_file_creation", "file_number": 4998, "file_size": 255513, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 80, "index_size": 253376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3585600, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 149400, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZeroLengthBlobStore", "creation_time": 1732000994, "oldest_key_time": 0, "file_creation_time": 1732000994, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4998, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:14.883102 139851137550080 [compaction_job.cc:703] job-00192: subcompact[0], size: 0.000256 G, files: /storage/mytopling/.rocksdb [004998] 2024/11/19-15:23:14.883685 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777216 2024/11/19-15:23:14.883797 139851137550080 (Original Log Time 2024/11/19-15:23:14.820572) EVENT_LOG_v1 {"time_micros": 1732000994820554, "job": 192, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4963], "score": 2.53482, "input_data_size": 685094, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:14.883803 139851137550080 (Original Log Time 2024/11/19-15:23:14.883373) [compaction_job.cc:2022] [default] [JOB 192] Compacted 1@2 files to L3 => 255513 bytes 2024/11/19-15:23:14.883806 139851137550080 (Original Log Time 2024/11/19-15:23:14.883658) [compaction_job.cc:1224] [default] compacted to: files[6 51 74 37 0 0 0] max score 2.53 (111 files need compaction), MB/sec: 11.0 rd, 4.1 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(0.7, 0.0 +0.0 blob) out(0.2 +0.0 blob), read-write-amplify(1.4) write-amplify(0.4) OK, records in: 149400, records dropped: 0 output_compression: Snappy 2024/11/19-15:23:14.883824 139851137550080 (Original Log Time 2024/11/19-15:23:14.883679) EVENT_LOG_v1 {"time_micros": 1732000994883668, "cf": "default", "job": 192, "event": "compaction_finished", "compaction_time_micros": 62539, "compaction_time_cpu_micros": 56344, "output_level": 3, "num_output_files": 1, "total_output_size": 255513, "num_input_records": 149400, "num_output_records": 149400, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 51, 74, 37, 0, 0, 0]} 2024/11/19-15:23:14.883892 139851137550080 [compaction_job.cc:2437] [default] [JOB 193] Compacting 1@2 files to L3, score 2.53, subcompactions 1 : 1 2024/11/19-15:23:14.883899 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 191 Base level 2, inputs: [4964(12MB)] 2024/11/19-15:23:15.164959 139851137550080 [dcompact_etcd.cc:1163] job-00193/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 21.053 MiB 2024/11/19-15:23:15.198469 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-01 2024/11/19-15:23:15.198552 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00191/att-01/compact.done) = No such file or directory: 0.889 sec on http probe failed: 1, retry 2024/11/19-15:23:15.605342 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-00 2024/11/19-15:23:15.605444 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00190/att-00/compact.done) = No such file or directory: 6.887 sec on http probe failed: 5, giveup 2024/11/19-15:23:15.605463 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00190/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 65.224 sec, real 7.206 (submit 312.982 ms wait 6.887 s), input 65.224 MiB, send shutdown 2024/11/19-15:23:15.615302 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-00 2024/11/19-15:23:15.615564 139851158521600 [dcompact_etcd.cc:1773] job-00190/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:15.639530 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-01 2024/11/19-15:23:15.639641 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00184/att-01/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:23:15.700136 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-00 2024/11/19-15:23:15.700216 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00193/att-00/compact.done) = No such file or directory: 0.526 sec on http probe failed: 1, retry 2024/11/19-15:23:15.752012 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-00 2024/11/19-15:23:15.752073 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00186/att-00/compact.done) = No such file or directory: 11.203 sec on http probe failed: 5, giveup 2024/11/19-15:23:15.752090 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00186/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.950 sec, real 11.477 (submit 266.645 ms wait 11.203 s), input 126.950 MiB, send shutdown 2024/11/19-15:23:15.761063 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-00 2024/11/19-15:23:15.761273 139851169007360 [dcompact_etcd.cc:1773] job-00186/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:15.899248 139851158521600 [dcompact_etcd.cc:1163] job-00190/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.284 sec, input = 65.224 MiB 2024/11/19-15:23:15.905084 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-02 2024/11/19-15:23:15.905142 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00183/att-02/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:23:15.905158 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00183/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.966 sec, real 11.503 (submit 290.852 ms wait 11.204 s), input 126.966 MiB, send shutdown 2024/11/19-15:23:15.912635 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00183/att-02 2024/11/19-15:23:15.912852 139851179493120 [dcompact_etcd.cc:1773] job-00183/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:15.913096 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00183) = success 2024/11/19-15:23:15.913103 139851179493120 [dcompact_etcd.cc:1773] job-00183/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:15.913121 139851179493120 (Original Log Time 2024/11/19-15:22:41.419386) EVENT_LOG_v1 {"time_micros": 1732000961419378, "job": 183, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4867, 4868, 4870, 4871], "score": 3.17235, "input_data_size": 134099008, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:15.913129 139851179493120 [compaction_job.cc:2437] [default] [JOB 183] Compacting 4@1 files to L2, score 3.17, subcompactions 1 : 1 2024/11/19-15:23:15.913136 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 181 Base level 1, inputs: [4867(31MB) 4868(31MB) 4870(31MB) 4871(31MB)] 2024/11/19-15:23:15.913361 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/004999.sst) /storage/mytopling/..//tmp/Topling-82038-46vLEv 2024/11/19-15:23:15.916951 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-01 2024/11/19-15:23:15.917015 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00191/att-01/compact.done) = No such file or directory: 1.610 sec on http probe failed: 2, retry 2024/11/19-15:23:15.968067 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004999.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:23:15.971428 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004999.sst): index pass time = 0.00's, 155.551'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64718 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:23:16.036083 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004999.sst): first pass time = 0.12's, 546.897'MB/sec 2024/11/19-15:23:16.036136 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004999.sst): old prealloc_size = 73819750, real_size = 66152592 2024/11/19-15:23:16.036192 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/004999.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64718 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.96 avg-zval =1020.21 2024/11/19-15:23:16.036259 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/004999.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.068 sec, Key+Value = 0.067 GB 2024/11/19-15:23:16.059618 139851169007360 [dcompact_etcd.cc:1163] job-00186/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.298 sec, input = 126.950 MiB 2024/11/19-15:23:16.096778 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/004999.sst): second pass time = 0.00's,183330075.209'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1090.863'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2100.961'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.373'MB/sec write SST data time = 0.02's, 3840.273'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 183330075.209'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64718 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.96 avg-zval = 1020.21 usrkeys = 64718 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.96 avg-zval = 1020.21 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8063 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0468 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 10.715015 GB avg = 0.256 KB (by entry num) total key len = 1.081227 GB avg = 0.026 KB total ukey len = 0.746041 GB avg = 0.018 KB total ukey num = 0.041898286 Billion total entry num = 0.041898286 Billion write speed all = 42.260957671 MB/sec (with seq num) write speed all = 41.060126662 MB/sec (without seq num) 2024/11/19-15:23:16.100380 139851179493120 [compaction_job.cc:1948] [default] [JOB 183] Generated table #4999: 64718 keys, 66137796 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:16.100431 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000996100407, "cf_name": "default", "job": 183, "event": "table_file_creation", "file_number": 4999, "file_size": 66137796, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66026048, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294360, "raw_average_key_size": 20, "raw_value_size": 65815497, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64718, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732000995, "oldest_key_time": 0, "file_creation_time": 1732000995, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 4999, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:16.115740 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005000.sst) /storage/mytopling/..//tmp/Topling-82038-grjyeC 2024/11/19-15:23:16.130395 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-00 2024/11/19-15:23:16.130487 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00193/att-00/compact.done) = No such file or directory: 0.956 sec on http probe failed: 2, retry 2024/11/19-15:23:16.168989 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005000.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:23:16.169110 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005000.sst): index pass time = 0.00's,5328.479'MB/sec index type = UintIndex_AllOne usrkeys = 63670 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:23:16.235526 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005000.sst): first pass time = 0.12's, 551.257'MB/sec 2024/11/19-15:23:16.235586 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005000.sst): old prealloc_size = 73819750, real_size = 64973704 2024/11/19-15:23:16.235609 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005000.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63670 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0648 GB zip-val = 0.0650 GB avg-val =1016.97 avg-zval =1020.22 2024/11/19-15:23:16.289695 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005000.sst): second pass time = 0.00's,82484481.529'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1200.671'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.984'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.410'MB/sec write SST data time = 0.02's, 4247.846'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 82484481.529'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63670 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.97 avg-zval = 1020.22 usrkeys = 63670 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.97 avg-zval = 1020.22 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0648 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0000 GB value = 0.0650 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index =3979.3750 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 10.779765 GB avg = 0.257 KB (by entry num) total key len = 1.082501 GB avg = 0.026 KB total ukey len = 0.746805 GB avg = 0.018 KB total ukey num = 0.041961956 Billion total entry num = 0.041961956 Billion write speed all = 42.468141297 MB/sec (with seq num) write speed all = 41.266316098 MB/sec (without seq num) 2024/11/19-15:23:16.293398 139851179493120 [compaction_job.cc:1948] [default] [JOB 183] Generated table #5000: 63670 keys, 64958794 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:16.293450 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000996293426, "cf_name": "default", "job": 183, "event": "table_file_creation", "file_number": 5000, "file_size": 64958794, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64957456, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1273400, "raw_average_key_size": 20, "raw_value_size": 64750318, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63670, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732000996, "oldest_key_time": 0, "file_creation_time": 1732000996, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5000, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:16.308342 139851179493120 [compaction_job.cc:703] job-00183: subcompact[0], size: 0.131097 G, files: /storage/mytopling/.rocksdb [004999,005000] 2024/11/19-15:23:16.309063 139851179493120 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777216 2024/11/19-15:23:16.309190 139851179493120 (Original Log Time 2024/11/19-15:23:15.913164) EVENT_LOG_v1 {"time_micros": 1732000995913148, "job": 183, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4867, 4868, 4870, 4871], "score": 3.17235, "input_data_size": 134099008, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:16.309197 139851179493120 (Original Log Time 2024/11/19-15:23:16.308741) [compaction_job.cc:2022] [default] [JOB 183] Compacted 4@1 files to L2 => 131096590 bytes 2024/11/19-15:23:16.309200 139851179493120 (Original Log Time 2024/11/19-15:23:16.309035) [compaction_job.cc:1224] [default] compacted to: files[6 47 76 37 0 0 0] max score 2.60 (112 files need compaction), MB/sec: 339.3 rd, 331.7 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128388, records dropped: 0 output_compression: Snappy 2024/11/19-15:23:16.309216 139851179493120 (Original Log Time 2024/11/19-15:23:16.309057) EVENT_LOG_v1 {"time_micros": 1732000996309046, "cf": "default", "job": 183, "event": "compaction_finished", "compaction_time_micros": 395190, "compaction_time_cpu_micros": 392204, "output_level": 2, "num_output_files": 2, "total_output_size": 131096590, "num_input_records": 128388, "num_output_records": 128388, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 47, 76, 37, 0, 0, 0]} 2024/11/19-15:23:16.314336 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004825.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:16.314358 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000996314356, "job": 183, "event": "table_file_deletion", "file_number": 4825} 2024/11/19-15:23:16.319400 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004828.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:16.319420 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000996319418, "job": 183, "event": "table_file_deletion", "file_number": 4828} 2024/11/19-15:23:16.325067 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004830.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:16.325087 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000996325085, "job": 183, "event": "table_file_deletion", "file_number": 4830} 2024/11/19-15:23:16.329712 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004832.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:16.329731 139851179493120 EVENT_LOG_v1 {"time_micros": 1732000996329729, "job": 183, "event": "table_file_deletion", "file_number": 4832} 2024/11/19-15:23:16.329877 139851179493120 [compaction_job.cc:2437] [default] [JOB 194] Compacting 4@2 files to L3, score 2.60, subcompactions 1 : 1 2024/11/19-15:23:16.329886 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 192 Base level 2, inputs: [4968(52MB) 4969(52MB) 4970(52MB) 4971(52MB)] 2024/11/19-15:23:16.544562 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-00 2024/11/19-15:23:16.544663 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00189/att-00/compact.done) = No such file or directory: 9.015 sec on http probe failed: 3, retry 2024/11/19-15:23:16.560351 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-00 2024/11/19-15:23:16.560411 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00193/att-00/compact.done) = No such file or directory: 1.387 sec on http probe failed: 3, retry 2024/11/19-15:23:16.614282 139851179493120 [dcompact_etcd.cc:1163] job-00194/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.284 sec, input = 255.642 MiB 2024/11/19-15:23:16.636463 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-01 2024/11/19-15:23:16.636525 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00191/att-01/compact.done) = No such file or directory: 2.328 sec on http probe failed: 3, retry 2024/11/19-15:23:16.641550 139852349699840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16273899 2024/11/19-15:23:16.986246 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-00 2024/11/19-15:23:16.986316 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00193/att-00/compact.done) = No such file or directory: 1.817 sec on http probe failed: 4, retry 2024/11/19-15:23:17.352783 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-01 2024/11/19-15:23:17.352880 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00191/att-01/compact.done) = No such file or directory: 3.048 sec on http probe failed: 4, retry 2024/11/19-15:23:17.412627 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-00 2024/11/19-15:23:17.412717 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00193/att-00/compact.done) = No such file or directory: 2.242 sec on http probe failed: 5, giveup 2024/11/19-15:23:17.412734 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00193/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 21.053 sec, real 2.528 (submit 280.659 ms wait 2.242 s), input 21.053 MiB, send shutdown 2024/11/19-15:23:17.423021 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-00 2024/11/19-15:23:17.423246 139851137550080 [dcompact_etcd.cc:1773] job-00193/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:17.536176 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-01 2024/11/19-15:23:17.536281 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00190/att-01/compact.done) = No such file or directory: 1.631 sec on http probe failed: 1, retry 2024/11/19-15:23:17.648709 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-01 2024/11/19-15:23:17.648809 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00184/att-01/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:23:17.717236 139851137550080 [dcompact_etcd.cc:1163] job-00193/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.294 sec, input = 21.053 MiB 2024/11/19-15:23:18.072819 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-01 2024/11/19-15:23:18.072935 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00191/att-01/compact.done) = No such file or directory: 3.764 sec on http probe failed: 5, giveup 2024/11/19-15:23:18.072957 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00191/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 35.565 sec, real 4.062 (submit 288.921 ms wait 3.764 s), input 35.565 MiB, send shutdown 2024/11/19-15:23:18.081789 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-01 2024/11/19-15:23:18.082060 139851148035840 [dcompact_etcd.cc:1773] job-00191/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:18.248790 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-01 2024/11/19-15:23:18.248870 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00193/att-01/compact.done) = No such file or directory: 0.526 sec on http probe failed: 1, retry 2024/11/19-15:23:18.359794 139851148035840 [dcompact_etcd.cc:1163] job-00191/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.278 sec, input = 35.565 MiB 2024/11/19-15:23:18.553931 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-00 2024/11/19-15:23:18.554010 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00189/att-00/compact.done) = No such file or directory: 11.023 sec on http probe failed: 4, retry 2024/11/19-15:23:18.675019 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-01 2024/11/19-15:23:18.675090 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00193/att-01/compact.done) = No such file or directory: 0.953 sec on http probe failed: 2, retry 2024/11/19-15:23:18.850153 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-01 2024/11/19-15:23:18.850260 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00190/att-01/compact.done) = No such file or directory: 2.942 sec on http probe failed: 2, retry 2024/11/19-15:23:19.101851 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-01 2024/11/19-15:23:19.101957 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00193/att-01/compact.done) = No such file or directory: 1.379 sec on http probe failed: 3, retry 2024/11/19-15:23:19.238860 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-01 2024/11/19-15:23:19.238976 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00186/att-01/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:23:19.253995 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-02 2024/11/19-15:23:19.254089 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00191/att-02/compact.done) = No such file or directory: 0.889 sec on http probe failed: 1, retry 2024/11/19-15:23:19.532706 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-01 2024/11/19-15:23:19.532782 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00193/att-01/compact.done) = No such file or directory: 1.806 sec on http probe failed: 4, retry 2024/11/19-15:23:19.656959 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-01 2024/11/19-15:23:19.657058 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00184/att-01/compact.done) = No such file or directory: 11.205 sec on http probe failed: 5, giveup 2024/11/19-15:23:19.657085 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00184/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.899 sec, real 11.490 (submit 276.448 ms wait 11.205 s), input 126.899 MiB, send shutdown 2024/11/19-15:23:19.662017 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-01 2024/11/19-15:23:19.662270 139850978166528 [dcompact_etcd.cc:1773] job-00184/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:19.965397 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-01 2024/11/19-15:23:19.965508 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00193/att-01/compact.done) = No such file or directory: 2.237 sec on http probe failed: 5, giveup 2024/11/19-15:23:19.965527 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00193/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 21.053 sec, real 2.542 (submit 293.919 ms wait 2.237 s), input 21.053 MiB, send shutdown 2024/11/19-15:23:19.970668 139850978166528 [dcompact_etcd.cc:1163] job-00184/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.308 sec, input = 126.899 MiB 2024/11/19-15:23:19.973935 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-01 2024/11/19-15:23:19.973953 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-02 2024/11/19-15:23:19.974028 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00191/att-02/compact.done) = No such file or directory: 1.606 sec on http probe failed: 2, retry 2024/11/19-15:23:19.974171 139851137550080 [dcompact_etcd.cc:1773] job-00193/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:20.164573 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-01 2024/11/19-15:23:20.164651 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00190/att-01/compact.done) = No such file or directory: 4.256 sec on http probe failed: 3, retry 2024/11/19-15:23:20.263471 139851137550080 [dcompact_etcd.cc:1163] job-00193/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 21.053 MiB 2024/11/19-15:23:20.563347 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-00 2024/11/19-15:23:20.563450 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00189/att-00/compact.done) = No such file or directory: 13.032 sec on http probe failed: 5, giveup 2024/11/19-15:23:20.563483 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00189/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 254.478 sec, real 13.362 (submit 320.016 ms wait 13.032 s), input 254.478 MiB, send shutdown 2024/11/19-15:23:20.569282 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-00 2024/11/19-15:23:20.569558 139851127064320 [dcompact_etcd.cc:1773] job-00189/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:20.693274 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-02 2024/11/19-15:23:20.693367 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00191/att-02/compact.done) = No such file or directory: 2.326 sec on http probe failed: 3, retry 2024/11/19-15:23:20.796166 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-02 2024/11/19-15:23:20.796259 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00193/att-02/compact.done) = No such file or directory: 0.526 sec on http probe failed: 1, retry 2024/11/19-15:23:20.863020 139851127064320 [dcompact_etcd.cc:1163] job-00189/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.293 sec, input = 254.478 MiB 2024/11/19-15:23:21.222093 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-02 2024/11/19-15:23:21.222191 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00193/att-02/compact.done) = No such file or directory: 0.954 sec on http probe failed: 2, retry 2024/11/19-15:23:21.247817 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-01 2024/11/19-15:23:21.247883 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00186/att-01/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:23:21.414737 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-02 2024/11/19-15:23:21.414832 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00191/att-02/compact.done) = No such file or directory: 3.045 sec on http probe failed: 4, retry 2024/11/19-15:23:21.474362 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-01 2024/11/19-15:23:21.474435 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00190/att-01/compact.done) = No such file or directory: 5.570 sec on http probe failed: 4, retry 2024/11/19-15:23:21.620129 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-00 2024/11/19-15:23:21.620201 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00194/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:23:21.651776 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-02 2024/11/19-15:23:21.651827 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00193/att-02/compact.done) = No such file or directory: 1.380 sec on http probe failed: 3, retry 2024/11/19-15:23:22.077897 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-02 2024/11/19-15:23:22.077999 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00193/att-02/compact.done) = No such file or directory: 1.809 sec on http probe failed: 4, retry 2024/11/19-15:23:22.135274 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-02 2024/11/19-15:23:22.135352 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00191/att-02/compact.done) = No such file or directory: 3.766 sec on http probe failed: 5, giveup 2024/11/19-15:23:22.135368 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00191/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 35.565 sec, real 4.053 (submit 277.658 ms wait 3.766 s), input 35.565 MiB, send shutdown 2024/11/19-15:23:22.143115 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00191/att-02 2024/11/19-15:23:22.143339 139851148035840 [dcompact_etcd.cc:1773] job-00191/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:22.145060 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00191) = success 2024/11/19-15:23:22.145068 139851148035840 [dcompact_etcd.cc:1773] job-00191/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:23:22.145087 139851148035840 (Original Log Time 2024/11/19-15:23:09.930637) EVENT_LOG_v1 {"time_micros": 1732000989930629, "job": 191, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4958, 4962], "score": 2.47538, "input_data_size": 30149994, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:22.145095 139851148035840 [compaction_job.cc:2437] [default] [JOB 191] Compacting 2@2 files to L3, score 2.48, subcompactions 1 : 1 2024/11/19-15:23:22.145102 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 189 Base level 2, inputs: [4958(12MB) 4962(16MB)] 2024/11/19-15:23:22.145340 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005001.sst) /storage/mytopling/..//tmp/Topling-82038-xgKVEB 2024/11/19-15:23:22.280546 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005001.sst): first pass time = 0.14's, 275.846'MB/sec 2024/11/19-15:23:22.280548 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005001.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.037 GB 2024/11/19-15:23:22.280590 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005001.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, dictZip workingMem = 0.0016 GB, waited 0.000 sec, Key+Value = 0.037 GB 2024/11/19-15:23:22.280675 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005001.sst): index pass time = 0.00's,43872.821'MB/sec index type = UintIndex_AllOne usrkeys = 504000 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0040 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:23:22.514957 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-02 2024/11/19-15:23:22.515077 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00193/att-02/compact.done) = No such file or directory: 2.236 sec on http probe failed: 5, giveup 2024/11/19-15:23:22.515103 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00193/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 21.053 sec, real 2.541 (submit 289.226 ms wait 2.236 s), input 21.053 MiB, send shutdown 2024/11/19-15:23:22.522053 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00193/att-02 2024/11/19-15:23:22.522326 139851137550080 [dcompact_etcd.cc:1773] job-00193/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:22.526136 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00193) = success 2024/11/19-15:23:22.526152 139851137550080 [dcompact_etcd.cc:1773] job-00193/att-03: CleanFiles 0.004 sec : 0.000 2024/11/19-15:23:22.526178 139851137550080 (Original Log Time 2024/11/19-15:23:14.883911) EVENT_LOG_v1 {"time_micros": 1732000994883905, "job": 193, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4964], "score": 2.5326, "input_data_size": 12776925, "oldest_snapshot_seqno": 139034989} 2024/11/19-15:23:22.526187 139851137550080 [compaction_job.cc:2437] [default] [JOB 193] Compacting 1@2 files to L3, score 2.53, subcompactions 1 : 1 2024/11/19-15:23:22.526198 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 191 Base level 2, inputs: [4964(12MB)] 2024/11/19-15:23:22.526544 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005002.sst) /storage/mytopling/..//tmp/Topling-82038-dFT1h0 2024/11/19-15:23:22.724527 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005002.sst: sumWaitingMem = 0.005 GB, sumWorkingMem = 0.002 GB, nltTrie workingMem = 0.0045 GB, waited 0.000 sec, Key+Value = 0.022 GB 2024/11/19-15:23:22.750441 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005002.sst): index pass time = 0.03's, 233.741'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 504000 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0060 GB zip-key = 0.0008 GB avg-key = 12.00 avg-zkey = 1.68 2024/11/19-15:23:22.762891 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005002.sst): first pass time = 0.24's, 93.410'MB/sec 2024/11/19-15:23:22.762945 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005002.sst): old prealloc_size = 14054617, real_size = 10993008 2024/11/19-15:23:22.763158 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005002.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 504000 prefix = 4 raw-key = 0.0060 GB zip-key = 0.0008 GB avg-key = 12.00 avg-zkey = 1.68 raw-val = 0.0100 GB zip-val = 0.0100 GB avg-val = 19.80 avg-zval = 19.88 2024/11/19-15:23:22.763212 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005002.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.002 GB, reorder workingMem = 0.0012 GB, waited 0.039 sec, Key+Value = 0.022 GB 2024/11/19-15:23:22.767203 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005001.sst): old prealloc_size = 33164993, real_size = 16754280 2024/11/19-15:23:22.767238 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005001.sst): index type = UintIndex_AllOne , store type = DictZipBlobStore usrkeys = 504000 prefix = 4 raw-key = 0.0040 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0272 GB zip-val = 0.0166 GB avg-val = 53.99 avg-zval = 32.99 2024/11/19-15:23:22.787165 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-01 2024/11/19-15:23:22.787247 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00190/att-01/compact.done) = No such file or directory: 6.880 sec on http probe failed: 5, giveup 2024/11/19-15:23:22.787268 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00190/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 65.224 sec, real 7.172 (submit 283.601 ms wait 6.880 s), input 65.224 MiB, send shutdown 2024/11/19-15:23:22.791647 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-01 2024/11/19-15:23:22.806186 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005001.sst): second pass time = 0.49's, 55.925'MB/sec, value only(73.0% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.04's, 433.738'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.662'MB/sec (index lex order gen) rebuild zvType time = 0.01's, 14.476'MB/sec write SST data time = 0.03's, 558.832'MB/sec dict compress time = 0.00's, 990.553'MB/sec z-dict build time = 0.00's, sample length = 0.273'MB, throughput = 5464.515'MB/sec zip my value time = 0.49's, unzip length = 0.027'GB zip my value throughput = 55.925'MB/sec zip pipeline throughput = 55.925'MB/sec entries = 504000 avg-key = 20.00 avg-zkey = 0.00 avg-val = 53.99 avg-zval = 32.99 usrkeys = 504000 avg-key = 12.00 avg-zkey = 0.00 avg-val = 53.99 avg-zval = 32.99 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 4 UnZipSize{ index = 0.0060 GB value = 0.0272 GB dict = 0.27 MB all = 0.0373 GB } __ZipSize{ index = 0.0000 GB value = 0.0166 GB dict = 0.27 MB all = 0.0169 GB } UnZip/Zip{ index =31500.0000 value = 1.6366 dict = 1.00 all = 2.2064 } Zip/UnZip{ index = 0.0000 value = 0.6110 dict = 1.00 all = 0.4532 } ---------------------------- total value len = 10.806978 GB avg = 0.254 KB (by entry num) total key len = 1.092581 GB avg = 0.026 KB total ukey len = 0.752853 GB avg = 0.018 KB total ukey num = 0.042465956 Billion total entry num = 0.042465956 Billion write speed all = 41.630426491 MB/sec (with seq num) write speed all = 40.441894432 MB/sec (without seq num) 2024/11/19-15:23:22.852351 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005002.sst): second pass time = 0.00's,44951351.351'MB/sec, value only(45.2% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 121.612'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 4038.267'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 2.660'MB/sec write SST data time = 0.04's, 260.010'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.010'GB zip my value throughput = 44951351.351'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 504000 avg-key = 24.00 avg-zkey = 1.68 avg-val = 19.80 avg-zval = 19.88 usrkeys = 504000 avg-key = 16.00 avg-zkey = 1.68 avg-val = 19.80 avg-zval = 19.88 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 1225 UnZipSize{ index = 0.0081 GB value = 0.0100 GB dict = 0.00 MB all = 0.0221 GB } __ZipSize{ index = 0.0008 GB value = 0.0100 GB dict = 0.00 MB all = 0.0109 GB } UnZip/Zip{ index = 9.5044 value = 0.9961 dict = 0.00 all = 2.0304 } Zip/UnZip{ index = 0.1052 value = 1.0039 dict = 0.00 all = 0.4925 } ---------------------------- total value len = 10.816957 GB avg = 0.252 KB (by entry num) total key len = 1.104677 GB avg = 0.026 KB total ukey len = 0.760917 GB avg = 0.018 KB total ukey num = 0.042969956 Billion total entry num = 0.042969956 Billion write speed all = 41.700921128 MB/sec (with seq num) write speed all = 40.498477384 MB/sec (without seq num) 2024/11/19-15:23:22.858206 139851158521600 [dcompact_etcd.cc:1773] job-00190/att-01: CleanFiles 0.067 sec : 0.000 2024/11/19-15:23:22.959851 139851148035840 [compaction_job.cc:1948] [default] [JOB 191] Generated table #5001: 504000 keys, 16902023 bytes, temperature: kUnknown 2024/11/19-15:23:22.959902 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001002959879, "cf_name": "default", "job": 191, "event": "table_file_creation", "file_number": 5001, "file_size": 16902023, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 16627952, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 10080000, "raw_average_key_size": 20, "raw_value_size": 27212584, "raw_average_value_size": 53, "num_data_blocks": 1, "num_entries": 504000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-DictZipBlobStore", "creation_time": 1732001002, "oldest_key_time": 0, "file_creation_time": 1732001002, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5001, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:22.966262 139851148035840 [compaction_job.cc:703] job-00191: subcompact[0], size: 0.016902 G, files: /storage/mytopling/.rocksdb [005001] 2024/11/19-15:23:22.969463 139851148035840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777215 2024/11/19-15:23:22.969588 139851148035840 (Original Log Time 2024/11/19-15:23:22.145130) EVENT_LOG_v1 {"time_micros": 1732001002145116, "job": 191, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4958, 4962], "score": 2.47538, "input_data_size": 30149994, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:22.969595 139851148035840 (Original Log Time 2024/11/19-15:23:22.969150) [compaction_job.cc:2022] [default] [JOB 191] Compacted 2@2 files to L3 => 16902023 bytes 2024/11/19-15:23:22.969598 139851148035840 (Original Log Time 2024/11/19-15:23:22.969422) [compaction_job.cc:1224] [default] compacted to: files[6 47 74 38 0 0 0] max score 2.44 (108 files need compaction), MB/sec: 36.7 rd, 20.6 wr, level 3, files in(2, 0) out(1 +0 blob) MB in(28.8, 0.0 +0.0 blob) out(16.1 +0.0 blob), read-write-amplify(1.6) write-amplify(0.6) OK, records in: 504000, records dropped: 0 output_compression: Snappy 2024/11/19-15:23:22.969616 139851148035840 (Original Log Time 2024/11/19-15:23:22.969456) EVENT_LOG_v1 {"time_micros": 1732001002969444, "cf": "default", "job": 191, "event": "compaction_finished", "compaction_time_micros": 821143, "compaction_time_cpu_micros": 499251, "output_level": 3, "num_output_files": 1, "total_output_size": 16902023, "num_input_records": 504000, "num_output_records": 504000, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 47, 74, 38, 0, 0, 0]} 2024/11/19-15:23:22.969688 139851148035840 [compaction_job.cc:2437] [default] [JOB 195] Compacting 4@2 files to L3, score 2.44, subcompactions 1 : 1 2024/11/19-15:23:22.969696 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 193 Base level 2, inputs: [4972(52MB) 4973(52MB) 4974(52MB) 4975(52MB)] 2024/11/19-15:23:23.061724 139851137550080 [compaction_job.cc:1948] [default] [JOB 193] Generated table #5002: 504000 keys, 10872209 bytes, temperature: kUnknown 2024/11/19-15:23:23.061780 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001003061755, "cf_name": "default", "job": 193, "event": "table_file_creation", "file_number": 5002, "file_size": 10872209, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 10018400, "index_size": 848448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12096000, "raw_average_key_size": 24, "raw_value_size": 9979200, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 504000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732001002, "oldest_key_time": 0, "file_creation_time": 1732001002, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5002, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:23.065503 139851137550080 [compaction_job.cc:703] job-00193: subcompact[0], size: 0.010872 G, files: /storage/mytopling/.rocksdb [005002] 2024/11/19-15:23:23.069588 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16777216 2024/11/19-15:23:23.069714 139851137550080 (Original Log Time 2024/11/19-15:23:22.526238) EVENT_LOG_v1 {"time_micros": 1732001002526215, "job": 193, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4964], "score": 2.5326, "input_data_size": 12776925, "oldest_snapshot_seqno": 139034989} 2024/11/19-15:23:23.069724 139851137550080 (Original Log Time 2024/11/19-15:23:23.069271) [compaction_job.cc:2022] [default] [JOB 193] Compacted 1@2 files to L3 => 10872209 bytes 2024/11/19-15:23:23.069727 139851137550080 (Original Log Time 2024/11/19-15:23:23.069557) [compaction_job.cc:1224] [default] compacted to: files[6 47 73 39 0 0 0] max score 2.42 (104 files need compaction), MB/sec: 23.7 rd, 20.2 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(12.2, 0.0 +0.0 blob) out(10.4 +0.0 blob), read-write-amplify(1.9) write-amplify(0.9) OK, records in: 504000, records dropped: 0 output_compression: Snappy 2024/11/19-15:23:23.069743 139851137550080 (Original Log Time 2024/11/19-15:23:23.069581) EVENT_LOG_v1 {"time_micros": 1732001003069568, "cf": "default", "job": 193, "event": "compaction_finished", "compaction_time_micros": 539277, "compaction_time_cpu_micros": 303509, "output_level": 3, "num_output_files": 1, "total_output_size": 10872209, "num_input_records": 504000, "num_output_records": 504000, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 47, 73, 39, 0, 0, 0]} 2024/11/19-15:23:23.069822 139851137550080 [compaction_job.cc:2437] [default] [JOB 196] Compacting 4@1 files to L2, score 2.42, subcompactions 1 : 1 2024/11/19-15:23:23.069831 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 194 Base level 1, inputs: [4885(31MB) 4887(31MB) 4888(31MB) 4889(31MB)] 2024/11/19-15:23:23.127979 139851158521600 [dcompact_etcd.cc:1163] job-00190/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.270 sec, input = 65.224 MiB 2024/11/19-15:23:23.151936 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-02 2024/11/19-15:23:23.152009 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00184/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:23:23.255733 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-01 2024/11/19-15:23:23.255817 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00186/att-01/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:23:23.274114 139851148035840 [dcompact_etcd.cc:1163] job-00195/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.304 sec, input = 255.640 MiB 2024/11/19-15:23:23.383209 139851137550080 [dcompact_etcd.cc:1163] job-00196/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.313 sec, input = 126.899 MiB 2024/11/19-15:23:23.628916 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-00 2024/11/19-15:23:23.629012 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00194/att-00/compact.done) = No such file or directory: 7.006 sec on http probe failed: 2, retry 2024/11/19-15:23:24.763930 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-02 2024/11/19-15:23:24.764015 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00190/att-02/compact.done) = No such file or directory: 1.631 sec on http probe failed: 1, retry 2024/11/19-15:23:25.161175 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-02 2024/11/19-15:23:25.161252 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00184/att-02/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:23:25.261556 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-01 2024/11/19-15:23:25.261623 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00186/att-01/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:23:25.637604 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-00 2024/11/19-15:23:25.637687 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00194/att-00/compact.done) = No such file or directory: 9.015 sec on http probe failed: 3, retry 2024/11/19-15:23:25.873312 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-01 2024/11/19-15:23:25.873415 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00189/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:23:26.077707 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-02 2024/11/19-15:23:26.077771 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00190/att-02/compact.done) = No such file or directory: 2.941 sec on http probe failed: 2, retry 2024/11/19-15:23:26.564779 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-00 2024/11/19-15:23:26.564866 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00196/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:23:27.170796 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-02 2024/11/19-15:23:27.170885 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00184/att-02/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:23:27.266309 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-01 2024/11/19-15:23:27.266376 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00186/att-01/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:23:27.266394 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00186/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.950 sec, real 11.505 (submit 298.265 ms wait 11.202 s), input 126.950 MiB, send shutdown 2024/11/19-15:23:27.271456 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-01 2024/11/19-15:23:27.271669 139851169007360 [dcompact_etcd.cc:1773] job-00186/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:27.388580 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-02 2024/11/19-15:23:27.388637 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00190/att-02/compact.done) = No such file or directory: 4.254 sec on http probe failed: 3, retry 2024/11/19-15:23:27.567030 139851169007360 [dcompact_etcd.cc:1163] job-00186/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.295 sec, input = 126.950 MiB 2024/11/19-15:23:27.646662 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-00 2024/11/19-15:23:27.646734 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00194/att-00/compact.done) = No such file or directory: 11.023 sec on http probe failed: 4, retry 2024/11/19-15:23:27.882995 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-01 2024/11/19-15:23:27.883067 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00189/att-01/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:23:28.282938 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-00 2024/11/19-15:23:28.283023 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00195/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:23:28.573728 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-00 2024/11/19-15:23:28.573804 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00196/att-00/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:23:28.701388 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-02 2024/11/19-15:23:28.701453 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00190/att-02/compact.done) = No such file or directory: 5.565 sec on http probe failed: 4, retry 2024/11/19-15:23:29.178442 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-02 2024/11/19-15:23:29.178526 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00184/att-02/compact.done) = No such file or directory: 9.200 sec on http probe failed: 4, retry 2024/11/19-15:23:29.532994 139852349699840 [db_impl_write.cc:2181] [default] New memtable created with log file: #5003. Immutable memtables: 0. 2024/11/19-15:23:29.533439 139852349699840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 16273899 2024/11/19-15:23:29.533539 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 197] Syncing log #4984 2024/11/19-15:23:29.533633 139850967680768 (Original Log Time 2024/11/19-15:23:29.533524) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:23:29.533641 139850967680768 [flush_job.cc:879] [default] [JOB 197] Flushing memtable with next log file: 5003 2024/11/19-15:23:29.533663 139850967680768 EVENT_LOG_v1 {"time_micros": 1732001009533656, "job": 197, "event": "flush_started", "num_memtables": 1, "num_entries": 1151675, "num_deletes": 0, "total_data_size": 473524361, "memory_usage": 539491720, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:23:29.533667 139850967680768 [flush_job.cc:910] [default] [JOB 197] Level-0 flush table #5004: started 2024/11/19-15:23:29.534012 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000024.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/005004.sst in ConvertToSST, mem_size = 514.499 M, time = 337.240 us 2024/11/19-15:23:29.651958 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-00 2024/11/19-15:23:29.652081 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00194/att-00/compact.done) = No such file or directory: 13.033 sec on http probe failed: 5, giveup 2024/11/19-15:23:29.652112 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00194/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.642 sec, real 13.322 (submit 283.968 ms wait 13.033 s), input 255.642 MiB, send shutdown 2024/11/19-15:23:29.658318 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-00 2024/11/19-15:23:29.679138 139851179493120 [dcompact_etcd.cc:1773] job-00194/att-00: CleanFiles 0.021 sec : 0.000 2024/11/19-15:23:29.887804 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-01 2024/11/19-15:23:29.887878 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00189/att-01/compact.done) = No such file or directory: 9.020 sec on http probe failed: 3, retry 2024/11/19-15:23:29.972091 139851179493120 [dcompact_etcd.cc:1163] job-00194/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.293 sec, input = 255.642 MiB 2024/11/19-15:23:30.015157 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-02 2024/11/19-15:23:30.015290 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00190/att-02/compact.done) = No such file or directory: 6.878 sec on http probe failed: 5, giveup 2024/11/19-15:23:30.015345 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00190/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 65.224 sec, real 7.157 (submit 269.682 ms wait 6.878 s), input 65.224 MiB, send shutdown 2024/11/19-15:23:30.023582 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00190/att-02 2024/11/19-15:23:30.060383 139851158521600 [dcompact_etcd.cc:1773] job-00190/att-02: CleanFiles 0.037 sec : 0.000 2024/11/19-15:23:30.060692 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/005004.sst): fsize = 514.501 M, time(ms): open: 0.034, seek: 0.014, finish: 0.035, meta: 0.014, Flush: 0.013, sync: 526.547, close: 0.009, all: 526.666 2024/11/19-15:23:30.061764 139850967680768 [flush_job.cc:1083] [default] [JOB 197] Flush lasted 528130 microseconds, and 232462 cpu microseconds. 2024/11/19-15:23:30.062228 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00190) = success 2024/11/19-15:23:30.062237 139851158521600 [dcompact_etcd.cc:1773] job-00190/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:23:30.062256 139851158521600 (Original Log Time 2024/11/19-15:23:08.398953) EVENT_LOG_v1 {"time_micros": 1732000988398947, "job": 190, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4955, 4957], "score": 2.43527, "input_data_size": 66709491, "oldest_snapshot_seqno": 138787056} 2024/11/19-15:23:30.062263 139851158521600 [compaction_job.cc:2437] [default] [JOB 190] Compacting 2@2 files to L3, score 2.44, subcompactions 1 : 1 2024/11/19-15:23:30.062271 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 188 Base level 2, inputs: [4955(62MB) 4957(1217KB)] 2024/11/19-15:23:30.082217 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005005.sst) /storage/mytopling/..//tmp/Topling-82038-LCWmXL 2024/11/19-15:23:30.082377 139850967680768 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 15785682 2024/11/19-15:23:30.082450 139850967680768 (Original Log Time 2024/11/19-15:23:30.060734) [flush_job.cc:1029] [default] [JOB 197] Level-0 flush table #5004: 539493146 bytes OK (needs compaction) 2024/11/19-15:23:30.082458 139850967680768 (Original Log Time 2024/11/19-15:23:30.061779) [memtable_list.cc:560] [default] Level-0 commit table #5004 started 2024/11/19-15:23:30.082463 139850967680768 (Original Log Time 2024/11/19-15:23:30.082350) [memtable_list.cc:759] [default] Level-0 commit table #5004: memtable #1 done 2024/11/19-15:23:30.082466 139850967680768 (Original Log Time 2024/11/19-15:23:30.082372) EVENT_LOG_v1 {"time_micros": 1732001010082366, "job": 197, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [7, 47, 73, 39, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:23:30.082470 139850967680768 (Original Log Time 2024/11/19-15:23:30.082421) [db_impl_compaction_flush.cc:310] [default] Level summary: files[7 47 73 39 0 0 0] max score 2.42 (101 files need compaction) 2024/11/19-15:23:30.149785 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005005.sst): first pass time = 0.07's,1012.414'MB/sec 2024/11/19-15:23:30.149788 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005005.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.068 GB 2024/11/19-15:23:30.149819 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005005.sst: sumWaitingMem = 0.004 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0041 GB, waited 0.000 sec, Key+Value = 0.068 GB 2024/11/19-15:23:30.155243 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005005.sst): index pass time = 0.01's, 226.323'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 102228 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:23:30.291830 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-00 2024/11/19-15:23:30.291905 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00195/att-00/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:23:30.591123 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-00 2024/11/19-15:23:30.591223 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00196/att-00/compact.done) = No such file or directory: 7.192 sec on http probe failed: 3, retry 2024/11/19-15:23:30.755256 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-02 2024/11/19-15:23:30.755388 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00186/att-02/compact.done) = No such file or directory: 3.176 sec on http probe failed: 1, retry 2024/11/19-15:23:31.195684 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-02 2024/11/19-15:23:31.195801 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00184/att-02/compact.done) = No such file or directory: 11.211 sec on http probe failed: 5, giveup 2024/11/19-15:23:31.195828 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00184/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.899 sec, real 11.533 (submit 308.295 ms wait 11.211 s), input 126.899 MiB, send shutdown 2024/11/19-15:23:31.206757 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00184/att-02 2024/11/19-15:23:31.207056 139850978166528 [dcompact_etcd.cc:1773] job-00184/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:31.208969 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00184) = success 2024/11/19-15:23:31.208984 139850978166528 [dcompact_etcd.cc:1773] job-00184/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:23:31.209010 139850978166528 (Original Log Time 2024/11/19-15:22:56.651049) EVENT_LOG_v1 {"time_micros": 1732000976651042, "job": 184, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4873, 4875, 4877, 4878], "score": 2.92257, "input_data_size": 134099648, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:31.209021 139850978166528 [compaction_job.cc:2437] [default] [JOB 184] Compacting 4@1 files to L2, score 2.92, subcompactions 1 : 1 2024/11/19-15:23:31.209032 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 182 Base level 1, inputs: [4873(31MB) 4875(31MB) 4877(31MB) 4878(31MB)] 2024/11/19-15:23:31.209451 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005006.sst) /storage/mytopling/..//tmp/Topling-82038-NeL7Q1 2024/11/19-15:23:31.309607 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005006.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.004 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:23:31.314391 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005006.sst): index pass time = 0.00's, 108.952'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64712 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:23:31.349427 139851158521600 [top_zip_table_builder.cc:2752] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005005.sst): part-0: reject ZipStore ratio = 0.803 2024/11/19-15:23:31.381361 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005006.sst): first pass time = 0.17's, 390.423'MB/sec 2024/11/19-15:23:31.381415 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005006.sst): old prealloc_size = 73819750, real_size = 66152512 2024/11/19-15:23:31.381469 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005006.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64712 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.05 avg-zval =1020.30 2024/11/19-15:23:31.381511 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005006.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.072 sec, Key+Value = 0.067 GB 2024/11/19-15:23:31.444049 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005006.sst): second pass time = 0.00's,215083117.647'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1056.189'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2197.436'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.361'MB/sec write SST data time = 0.02's, 3719.054'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 215083117.647'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64712 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.05 avg-zval = 1020.30 usrkeys = 64712 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.05 avg-zval = 1020.30 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0462 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 10.882773 GB avg = 0.253 KB (by entry num) total key len = 1.105971 GB avg = 0.026 KB total ukey len = 0.761693 GB avg = 0.018 KB total ukey num = 0.043034668 Billion total entry num = 0.043034668 Billion write speed all = 40.712141071 MB/sec (with seq num) write speed all = 39.543022069 MB/sec (without seq num) 2024/11/19-15:23:31.456526 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005005.sst): old prealloc_size = 73380440, real_size = 66245984 2024/11/19-15:23:31.456610 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005005.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 102228 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0659 GB zip-val = 0.0660 GB avg-val = 645.01 avg-zval = 646.07 2024/11/19-15:23:31.456656 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005005.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 1.307 sec, Key+Value = 0.068 GB 2024/11/19-15:23:31.569473 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005005.sst): second pass time = 1.20's, 54.967'MB/sec, value only(96.4% of KV) wait indexing time = 0.11's, remap KeyValue time = 0.11's, 586.368'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2517.548'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 0.284'MB/sec write SST data time = 0.02's, 2879.732'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = 0.000'MB/sec zip my value time = 1.20's, unzip length = 0.066'GB zip my value throughput = 54.967'MB/sec zip pipeline throughput = 54.967'MB/sec entries = 102228 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.01 avg-zval = 646.07 usrkeys = 102228 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.01 avg-zval = 646.07 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8055 UnZipSize{ index = 0.0016 GB value = 0.0659 GB dict = 0.00 MB all = 0.0684 GB } __ZipSize{ index = 0.0002 GB value = 0.0660 GB dict = 0.00 MB all = 0.0662 GB } UnZip/Zip{ index = 9.4341 value = 0.9984 dict = 0.00 all = 1.0328 } Zip/UnZip{ index = 0.1060 value = 1.0016 dict = 0.00 all = 0.9683 } ---------------------------- total value len = 10.948711 GB avg = 0.254 KB (by entry num) total key len = 1.108424 GB avg = 0.026 KB total ukey len = 0.763329 GB avg = 0.018 KB total ukey num = 0.043136896 Billion total entry num = 0.043136896 Billion write speed all = 40.926959061 MB/sec (with seq num) write speed all = 39.755561761 MB/sec (without seq num) 2024/11/19-15:23:31.703514 139850978166528 [compaction_job.cc:1948] [default] [JOB 184] Generated table #5006: 64712 keys, 66137732 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:31.703576 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001011703551, "cf_name": "default", "job": 184, "event": "table_file_creation", "file_number": 5006, "file_size": 66137732, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025968, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294240, "raw_average_key_size": 20, "raw_value_size": 65815434, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64712, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001011, "oldest_key_time": 0, "file_creation_time": 1732001011, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5006, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:31.720515 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005007.sst) /storage/mytopling/..//tmp/Topling-82038-Lmq0XG 2024/11/19-15:23:31.774310 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005007.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:23:31.777602 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005007.sst): index pass time = 0.00's, 156.185'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63601 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:23:31.840931 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005007.sst): first pass time = 0.12's, 547.824'MB/sec 2024/11/19-15:23:31.840984 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005007.sst): old prealloc_size = 73819750, real_size = 65013232 2024/11/19-15:23:31.841037 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005007.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63601 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1016.99 avg-zval =1020.25 2024/11/19-15:23:31.841077 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005007.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.066 GB 2024/11/19-15:23:31.896772 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-01 2024/11/19-15:23:31.896850 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00189/att-01/compact.done) = No such file or directory: 11.025 sec on http probe failed: 4, retry 2024/11/19-15:23:31.901046 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005007.sst): second pass time = 0.00's,373883739.884'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1082.430'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2185.133'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.377'MB/sec write SST data time = 0.02's, 3651.057'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 373883739.884'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63601 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.25 usrkeys = 63601 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7925 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0438 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 11.013393 GB avg = 0.255 KB (by entry num) total key len = 1.109696 GB avg = 0.026 KB total ukey len = 0.764092 GB avg = 0.018 KB total ukey num = 0.043200497 Billion total entry num = 0.043200497 Billion write speed all = 41.104570457 MB/sec (with seq num) write speed all = 39.932764932 MB/sec (without seq num) 2024/11/19-15:23:32.008670 139851158521600 [compaction_job.cc:1948] [default] [JOB 190] Generated table #5005: 102228 keys, 66222353 bytes, temperature: kUnknown 2024/11/19-15:23:32.008726 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001012008701, "cf_name": "default", "job": 190, "event": "table_file_creation", "file_number": 5005, "file_size": 66222353, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66046864, "index_size": 173376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2453472, "raw_average_key_size": 24, "raw_value_size": 65938423, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 102228, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732001010, "oldest_key_time": 0, "file_creation_time": 1732001010, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5005, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:32.085940 139850978166528 [compaction_job.cc:1948] [default] [JOB 184] Generated table #5007: 63601 keys, 64998724 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:32.085984 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001012085964, "cf_name": "default", "job": 184, "event": "table_file_creation", "file_number": 5007, "file_size": 64998724, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64888800, "index_size": 108352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1272020, "raw_average_key_size": 20, "raw_value_size": 64681887, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63601, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001011, "oldest_key_time": 0, "file_creation_time": 1732001011, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5007, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:32.102712 139850978166528 [compaction_job.cc:703] job-00184: subcompact[0], size: 0.131136 G, files: /storage/mytopling/.rocksdb [005006,005007] 2024/11/19-15:23:32.103477 139850978166528 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 16273898 2024/11/19-15:23:32.103632 139850978166528 (Original Log Time 2024/11/19-15:23:31.209174) EVENT_LOG_v1 {"time_micros": 1732001011209049, "job": 184, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4873, 4875, 4877, 4878], "score": 2.92257, "input_data_size": 134099648, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:32.103638 139850978166528 (Original Log Time 2024/11/19-15:23:32.103142) [compaction_job.cc:2022] [default] [JOB 184] Compacted 4@1 files to L2 => 131136456 bytes 2024/11/19-15:23:32.103641 139850978166528 (Original Log Time 2024/11/19-15:23:32.103445) [compaction_job.cc:1224] [default] compacted to: files[7 43 75 39 0 0 0] max score 2.64 (103 files need compaction), MB/sec: 150.1 rd, 146.8 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.1 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128313, records dropped: 0 output_compression: Snappy 2024/11/19-15:23:32.103659 139850978166528 (Original Log Time 2024/11/19-15:23:32.103470) EVENT_LOG_v1 {"time_micros": 1732001012103457, "cf": "default", "job": 184, "event": "compaction_finished", "compaction_time_micros": 893550, "compaction_time_cpu_micros": 428877, "output_level": 2, "num_output_files": 2, "total_output_size": 131136456, "num_input_records": 128313, "num_output_records": 128313, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [7, 43, 75, 39, 0, 0, 0]} 2024/11/19-15:23:32.134622 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004835.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:32.134647 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001012134645, "job": 184, "event": "table_file_deletion", "file_number": 4835} 2024/11/19-15:23:32.162344 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004837.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:32.162366 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001012162364, "job": 184, "event": "table_file_deletion", "file_number": 4837} 2024/11/19-15:23:32.194510 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004839.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:32.194531 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001012194529, "job": 184, "event": "table_file_deletion", "file_number": 4839} 2024/11/19-15:23:32.218888 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004842.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:32.218908 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001012218906, "job": 184, "event": "table_file_deletion", "file_number": 4842} 2024/11/19-15:23:32.219080 139850978166528 [compaction_job.cc:2437] [default] [JOB 198] Compacting 3@2 files to L3, score 2.35, subcompactions 1 : 1 2024/11/19-15:23:32.219089 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 196 Base level 2, inputs: [4978(52MB) 4979(52MB) 4967(46MB)] 2024/11/19-15:23:32.300786 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-00 2024/11/19-15:23:32.300859 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00195/att-00/compact.done) = No such file or directory: 9.018 sec on http probe failed: 3, retry 2024/11/19-15:23:32.514273 139850978166528 [dcompact_etcd.cc:1163] job-00198/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.295 sec, input = 184.525 MiB 2024/11/19-15:23:32.596071 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-00 2024/11/19-15:23:32.596133 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00196/att-00/compact.done) = No such file or directory: 9.208 sec on http probe failed: 4, retry 2024/11/19-15:23:32.761580 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-02 2024/11/19-15:23:32.761643 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00186/att-02/compact.done) = No such file or directory: 5.188 sec on http probe failed: 2, retry 2024/11/19-15:23:33.150705 139851158521600 [compaction_job.cc:703] job-00190: subcompact[0], size: 0.066222 G, files: /storage/mytopling/.rocksdb [005005] 2024/11/19-15:23:33.151372 139851158521600 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 16777214 2024/11/19-15:23:33.151487 139851158521600 (Original Log Time 2024/11/19-15:23:30.062307) EVENT_LOG_v1 {"time_micros": 1732001010062283, "job": 190, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4955, 4957], "score": 2.43527, "input_data_size": 66709491, "oldest_snapshot_seqno": 138787056} 2024/11/19-15:23:33.151494 139851158521600 (Original Log Time 2024/11/19-15:23:33.151060) [compaction_job.cc:2022] [default] [JOB 190] Compacted 2@2 files to L3 => 66222353 bytes 2024/11/19-15:23:33.151497 139851158521600 (Original Log Time 2024/11/19-15:23:33.151337) [compaction_job.cc:1224] [default] compacted to: files[7 43 73 40 0 0 0] max score 2.64 (100 files need compaction), MB/sec: 21.6 rd, 21.4 wr, level 3, files in(2, 0) out(1 +0 blob) MB in(63.6, 0.0 +0.0 blob) out(63.2 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 102228, records dropped: 0 output_compression: Snappy 2024/11/19-15:23:33.151513 139851158521600 (Original Log Time 2024/11/19-15:23:33.151364) EVENT_LOG_v1 {"time_micros": 1732001013151351, "cf": "default", "job": 190, "event": "compaction_finished", "compaction_time_micros": 3088417, "compaction_time_cpu_micros": 536046, "output_level": 3, "num_output_files": 1, "total_output_size": 66222353, "num_input_records": 102228, "num_output_records": 102228, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [7, 43, 73, 40, 0, 0, 0]} 2024/11/19-15:23:33.151590 139851158521600 [compaction_job.cc:2437] [default] [JOB 199] Compacting 4@2 files to L3, score 2.23, subcompactions 1 : 1 2024/11/19-15:23:33.151598 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 197 Base level 2, inputs: [4607(62MB) 4608(63MB) 4609(61MB) 4610(63MB)] 2024/11/19-15:23:33.438845 139851158521600 [dcompact_etcd.cc:1163] job-00199/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.287 sec, input = 253.865 MiB 2024/11/19-15:23:33.901720 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-01 2024/11/19-15:23:33.901790 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00189/att-01/compact.done) = No such file or directory: 13.034 sec on http probe failed: 5, giveup 2024/11/19-15:23:33.901808 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00189/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 254.478 sec, real 13.332 (submit 293.344 ms wait 13.034 s), input 254.478 MiB, send shutdown 2024/11/19-15:23:33.909933 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-01 2024/11/19-15:23:33.910282 139851127064320 [dcompact_etcd.cc:1773] job-00189/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:34.205993 139851127064320 [dcompact_etcd.cc:1163] job-00189/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.296 sec, input = 254.478 MiB 2024/11/19-15:23:34.306392 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-00 2024/11/19-15:23:34.306458 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00195/att-00/compact.done) = No such file or directory: 11.027 sec on http probe failed: 4, retry 2024/11/19-15:23:34.604541 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-00 2024/11/19-15:23:34.604619 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00196/att-00/compact.done) = No such file or directory: 11.213 sec on http probe failed: 5, giveup 2024/11/19-15:23:34.604638 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00196/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.899 sec, real 11.534 (submit 312.928 ms wait 11.213 s), input 126.899 MiB, send shutdown 2024/11/19-15:23:34.611175 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-00 2024/11/19-15:23:34.611418 139851137550080 [dcompact_etcd.cc:1773] job-00196/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:34.770120 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-02 2024/11/19-15:23:34.770235 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00186/att-02/compact.done) = No such file or directory: 7.195 sec on http probe failed: 3, retry 2024/11/19-15:23:34.878137 139851137550080 [dcompact_etcd.cc:1163] job-00196/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.267 sec, input = 126.899 MiB 2024/11/19-15:23:34.980409 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-01 2024/11/19-15:23:34.980519 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00194/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:23:36.316236 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-00 2024/11/19-15:23:36.316316 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00195/att-00/compact.done) = No such file or directory: 13.032 sec on http probe failed: 5, giveup 2024/11/19-15:23:36.316335 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00195/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.640 sec, real 13.346 (submit 303.985 ms wait 13.032 s), input 255.640 MiB, send shutdown 2024/11/19-15:23:36.321139 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-00 2024/11/19-15:23:36.321394 139851148035840 [dcompact_etcd.cc:1773] job-00195/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:36.587192 139851148035840 [dcompact_etcd.cc:1163] job-00195/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.266 sec, input = 255.640 MiB 2024/11/19-15:23:36.775000 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-02 2024/11/19-15:23:36.775092 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00186/att-02/compact.done) = No such file or directory: 9.203 sec on http probe failed: 4, retry 2024/11/19-15:23:36.985501 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-01 2024/11/19-15:23:36.985617 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00194/att-01/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:23:37.132337 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-00 2024/11/19-15:23:37.132415 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00198/att-00/compact.done) = No such file or directory: 4.613 sec on http probe failed: 1, retry 2024/11/19-15:23:38.057063 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-01 2024/11/19-15:23:38.057178 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00196/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:23:38.447570 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-00 2024/11/19-15:23:38.447679 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00199/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:23:38.785511 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-02 2024/11/19-15:23:38.785591 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00186/att-02/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:23:38.785610 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00186/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.950 sec, real 11.514 (submit 295.288 ms wait 11.208 s), input 126.950 MiB, send shutdown 2024/11/19-15:23:38.791655 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00186/att-02 2024/11/19-15:23:38.791872 139851169007360 [dcompact_etcd.cc:1773] job-00186/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:38.792117 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00186) = success 2024/11/19-15:23:38.792124 139851169007360 [dcompact_etcd.cc:1773] job-00186/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:38.792150 139851169007360 (Original Log Time 2024/11/19-15:23:04.274198) EVENT_LOG_v1 {"time_micros": 1732000984274190, "job": 186, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4879, 4881, 4883, 4884], "score": 2.67279, "input_data_size": 134098496, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:38.792158 139851169007360 [compaction_job.cc:2437] [default] [JOB 186] Compacting 4@1 files to L2, score 2.67, subcompactions 1 : 1 2024/11/19-15:23:38.792165 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 184 Base level 1, inputs: [4879(31MB) 4881(31MB) 4883(31MB) 4884(31MB)] 2024/11/19-15:23:38.792371 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005008.sst) /storage/mytopling/..//tmp/Topling-82038-BQKCK7 2024/11/19-15:23:38.845714 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005008.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:23:38.845814 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005008.sst): index pass time = 0.00's,7320.588'MB/sec index type = UintIndex_AllOne usrkeys = 64714 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:23:38.912799 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005008.sst): first pass time = 0.12's, 557.318'MB/sec 2024/11/19-15:23:38.912850 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005008.sst): old prealloc_size = 73819750, real_size = 66042200 2024/11/19-15:23:38.912873 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005008.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64714 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.02 avg-zval =1020.27 2024/11/19-15:23:38.967865 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005008.sst): second pass time = 0.00's,150606775.744'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1200.579'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.176'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.417'MB/sec write SST data time = 0.02's, 4091.007'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 150606775.744'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64714 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.02 avg-zval = 1020.27 usrkeys = 64714 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.02 avg-zval = 1020.27 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.6250 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 11.079208 GB avg = 0.256 KB (by entry num) total key len = 1.110991 GB avg = 0.026 KB total ukey len = 0.764869 GB avg = 0.018 KB total ukey num = 0.043265211 Billion total entry num = 0.043265211 Billion write speed all = 40.364937435 MB/sec (with seq num) write speed all = 39.218837965 MB/sec (without seq num) 2024/11/19-15:23:38.971464 139851169007360 [compaction_job.cc:1948] [default] [JOB 186] Generated table #5008: 64714 keys, 66027018 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:38.971516 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001018971492, "cf_name": "default", "job": 186, "event": "table_file_creation", "file_number": 5008, "file_size": 66027018, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025696, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294280, "raw_average_key_size": 20, "raw_value_size": 65815161, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64714, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001018, "oldest_key_time": 0, "file_creation_time": 1732001018, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5008, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:38.986794 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005009.sst) /storage/mytopling/..//tmp/Topling-82038-RjHSPI 2024/11/19-15:23:38.996449 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-01 2024/11/19-15:23:38.996547 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00194/att-01/compact.done) = No such file or directory: 9.014 sec on http probe failed: 3, retry 2024/11/19-15:23:39.040106 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005009.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:23:39.043368 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005009.sst): index pass time = 0.00's, 157.929'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63653 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:23:39.106727 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005009.sst): first pass time = 0.12's, 550.449'MB/sec 2024/11/19-15:23:39.106785 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005009.sst): old prealloc_size = 73819750, real_size = 65065744 2024/11/19-15:23:39.106843 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005009.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63653 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1016.99 avg-zval =1020.24 2024/11/19-15:23:39.106890 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005009.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.066 GB 2024/11/19-15:23:39.145117 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-00 2024/11/19-15:23:39.145228 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00198/att-00/compact.done) = No such file or directory: 6.619 sec on http probe failed: 2, retry 2024/11/19-15:23:39.172353 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005009.sst): second pass time = 0.00's,158274224.939'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 992.341'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2010.944'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.336'MB/sec write SST data time = 0.02's, 3597.791'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 158274224.939'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63653 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 usrkeys = 63653 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7931 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0651 GB } UnZip/Zip{ index = 7.0454 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 11.143942 GB avg = 0.257 KB (by entry num) total key len = 1.112264 GB avg = 0.026 KB total ukey len = 0.765633 GB avg = 0.018 KB total ukey num = 0.043328864 Billion total entry num = 0.043328864 Billion write speed all = 40.556043357 MB/sec (with seq num) write speed all = 39.409034375 MB/sec (without seq num) 2024/11/19-15:23:39.178254 139851169007360 [compaction_job.cc:1948] [default] [JOB 186] Generated table #5009: 63653 keys, 65051204 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:39.178316 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001019178284, "cf_name": "default", "job": 186, "event": "table_file_creation", "file_number": 5009, "file_size": 65051204, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64941248, "index_size": 108416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1273060, "raw_average_key_size": 20, "raw_value_size": 64734158, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63653, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001018, "oldest_key_time": 0, "file_creation_time": 1732001018, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5009, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:39.194388 139851169007360 [compaction_job.cc:703] job-00186: subcompact[0], size: 0.131078 G, files: /storage/mytopling/.rocksdb [005008,005009] 2024/11/19-15:23:39.200143 139851169007360 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 16777216 2024/11/19-15:23:39.200283 139851169007360 (Original Log Time 2024/11/19-15:23:38.792196) EVENT_LOG_v1 {"time_micros": 1732001018792177, "job": 186, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4879, 4881, 4883, 4884], "score": 2.67279, "input_data_size": 134098496, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:39.200289 139851169007360 (Original Log Time 2024/11/19-15:23:39.199814) [compaction_job.cc:2022] [default] [JOB 186] Compacted 4@1 files to L2 => 131078222 bytes 2024/11/19-15:23:39.200293 139851169007360 (Original Log Time 2024/11/19-15:23:39.200096) [compaction_job.cc:1224] [default] compacted to: files[7 39 75 40 0 0 0] max score 2.92 (98 files need compaction), MB/sec: 333.4 rd, 325.9 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128367, records dropped: 0 output_compression: Snappy 2024/11/19-15:23:39.200309 139851169007360 (Original Log Time 2024/11/19-15:23:39.200136) EVENT_LOG_v1 {"time_micros": 1732001019200110, "cf": "default", "job": 186, "event": "compaction_finished", "compaction_time_micros": 402205, "compaction_time_cpu_micros": 397576, "output_level": 2, "num_output_files": 2, "total_output_size": 131078222, "num_input_records": 128367, "num_output_records": 128367, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [7, 39, 75, 40, 0, 0, 0]} 2024/11/19-15:23:39.215184 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-02 2024/11/19-15:23:39.215281 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00189/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:23:39.218506 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004845.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:39.218534 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001019218531, "job": 186, "event": "table_file_deletion", "file_number": 4845} 2024/11/19-15:23:39.237251 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004847.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:39.237278 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001019237275, "job": 186, "event": "table_file_deletion", "file_number": 4847} 2024/11/19-15:23:39.258264 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004848.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:39.258289 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001019258287, "job": 186, "event": "table_file_deletion", "file_number": 4848} 2024/11/19-15:23:39.276180 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004850.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:39.276204 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001019276202, "job": 186, "event": "table_file_deletion", "file_number": 4850} 2024/11/19-15:23:39.276374 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004937.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:39.276388 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001019276386, "job": 186, "event": "table_file_deletion", "file_number": 4937} 2024/11/19-15:23:39.276553 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004938.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:39.276565 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001019276563, "job": 186, "event": "table_file_deletion", "file_number": 4938} 2024/11/19-15:23:39.276763 139851169007360 [compaction_job.cc:2437] [default] [JOB 200] Compacting 4@2 files to L3, score 2.15, subcompactions 1 : 1 2024/11/19-15:23:39.276775 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 198 Base level 2, inputs: [4611(61MB) 4612(62MB) 4613(61MB) 4620(63MB)] 2024/11/19-15:23:39.567366 139851169007360 [dcompact_etcd.cc:1163] job-00200/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.290 sec, input = 253.783 MiB 2024/11/19-15:23:40.062935 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-01 2024/11/19-15:23:40.063031 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00196/att-01/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:23:40.453622 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-00 2024/11/19-15:23:40.453699 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00199/att-00/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:23:41.004410 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-01 2024/11/19-15:23:41.004506 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00194/att-01/compact.done) = No such file or directory: 11.025 sec on http probe failed: 4, retry 2024/11/19-15:23:41.150187 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-00 2024/11/19-15:23:41.150285 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00198/att-00/compact.done) = No such file or directory: 8.631 sec on http probe failed: 3, retry 2024/11/19-15:23:41.221041 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-02 2024/11/19-15:23:41.221115 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00189/att-02/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:23:41.597061 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-01 2024/11/19-15:23:41.597166 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00195/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:23:42.073836 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-01 2024/11/19-15:23:42.073942 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00196/att-01/compact.done) = No such file or directory: 7.185 sec on http probe failed: 3, retry 2024/11/19-15:23:42.459275 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-00 2024/11/19-15:23:42.459356 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00199/att-00/compact.done) = No such file or directory: 9.015 sec on http probe failed: 3, retry 2024/11/19-15:23:43.015551 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-01 2024/11/19-15:23:43.015658 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00194/att-01/compact.done) = No such file or directory: 13.032 sec on http probe failed: 5, giveup 2024/11/19-15:23:43.015678 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00194/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.642 sec, real 13.336 (submit 292.853 ms wait 13.032 s), input 255.642 MiB, send shutdown 2024/11/19-15:23:43.025959 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-01 2024/11/19-15:23:43.026198 139851179493120 [dcompact_etcd.cc:1773] job-00194/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:43.157544 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-00 2024/11/19-15:23:43.157614 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00198/att-00/compact.done) = No such file or directory: 10.636 sec on http probe failed: 4, retry 2024/11/19-15:23:43.229464 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-02 2024/11/19-15:23:43.229523 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00189/att-02/compact.done) = No such file or directory: 9.015 sec on http probe failed: 3, retry 2024/11/19-15:23:43.321485 139851179493120 [dcompact_etcd.cc:1163] job-00194/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.295 sec, input = 255.642 MiB 2024/11/19-15:23:43.605733 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-01 2024/11/19-15:23:43.605805 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00195/att-01/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:23:44.085839 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-01 2024/11/19-15:23:44.085917 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00196/att-01/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:23:44.467818 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-00 2024/11/19-15:23:44.467898 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00199/att-00/compact.done) = No such file or directory: 11.021 sec on http probe failed: 4, retry 2024/11/19-15:23:44.577207 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-00 2024/11/19-15:23:44.577289 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00200/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:23:44.660433 139852349699840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 16273899 2024/11/19-15:23:45.162676 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-00 2024/11/19-15:23:45.162753 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00198/att-00/compact.done) = No such file or directory: 12.643 sec on http probe failed: 5, giveup 2024/11/19-15:23:45.162772 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00198/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 184.525 sec, real 12.943 (submit 294.746 ms wait 12.643 s), input 184.525 MiB, send shutdown 2024/11/19-15:23:45.171391 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-00 2024/11/19-15:23:45.171608 139850978166528 [dcompact_etcd.cc:1773] job-00198/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:45.239143 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-02 2024/11/19-15:23:45.239203 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00189/att-02/compact.done) = No such file or directory: 11.024 sec on http probe failed: 4, retry 2024/11/19-15:23:45.442579 139850978166528 [dcompact_etcd.cc:1163] job-00198/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.271 sec, input = 184.525 MiB 2024/11/19-15:23:45.619752 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-01 2024/11/19-15:23:45.619829 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00195/att-01/compact.done) = No such file or directory: 9.019 sec on http probe failed: 3, retry 2024/11/19-15:23:46.095727 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-01 2024/11/19-15:23:46.095805 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00196/att-01/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:23:46.095825 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00196/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.899 sec, real 11.484 (submit 266.642 ms wait 11.208 s), input 126.899 MiB, send shutdown 2024/11/19-15:23:46.100315 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-01 2024/11/19-15:23:46.100527 139851137550080 [dcompact_etcd.cc:1773] job-00196/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:46.367255 139851137550080 [dcompact_etcd.cc:1163] job-00196/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.267 sec, input = 126.899 MiB 2024/11/19-15:23:46.480635 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-00 2024/11/19-15:23:46.480701 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00199/att-00/compact.done) = No such file or directory: 13.029 sec on http probe failed: 5, giveup 2024/11/19-15:23:46.480719 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00199/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.865 sec, real 13.329 (submit 286.814 ms wait 13.029 s), input 253.865 MiB, send shutdown 2024/11/19-15:23:46.486183 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-00 2024/11/19-15:23:46.486390 139851158521600 [dcompact_etcd.cc:1773] job-00199/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:46.582936 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-00 2024/11/19-15:23:46.583026 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00200/att-00/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:23:46.787295 139851158521600 [dcompact_etcd.cc:1163] job-00199/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.301 sec, input = 253.865 MiB 2024/11/19-15:23:47.244328 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-02 2024/11/19-15:23:47.244420 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00189/att-02/compact.done) = No such file or directory: 13.033 sec on http probe failed: 5, giveup 2024/11/19-15:23:47.244437 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00189/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 254.478 sec, real 13.334 (submit 295.638 ms wait 13.033 s), input 254.478 MiB, send shutdown 2024/11/19-15:23:47.252490 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00189/att-02 2024/11/19-15:23:47.252712 139851127064320 [dcompact_etcd.cc:1773] job-00189/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:47.253034 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00189) = success 2024/11/19-15:23:47.253041 139851127064320 [dcompact_etcd.cc:1773] job-00189/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:47.253061 139851127064320 (Original Log Time 2024/11/19-15:23:07.201556) EVENT_LOG_v1 {"time_micros": 1732000987201550, "job": 189, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4953, 4954, 4956, 4959], "score": 2.51838, "input_data_size": 259250868, "oldest_snapshot_seqno": 138767053} 2024/11/19-15:23:47.253069 139851127064320 [compaction_job.cc:2437] [default] [JOB 189] Compacting 4@2 files to L3, score 2.52, subcompactions 1 : 1 2024/11/19-15:23:47.253077 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 187 Base level 2, inputs: [4953(62MB) 4954(61MB) 4956(62MB) 4959(61MB)] 2024/11/19-15:23:47.253308 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005010.sst) /storage/mytopling/..//tmp/Topling-82038-Xo7cMB 2024/11/19-15:23:47.410664 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005010.sst): first pass time = 0.16's, 853.027'MB/sec 2024/11/19-15:23:47.410670 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005010.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0018 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:23:47.410701 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005010.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.002 GB, dictZip workingMem = 0.0081 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:23:47.421322 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005010.sst): index pass time = 0.01's, 226.703'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 200576 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 2024/11/19-15:23:47.627267 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-01 2024/11/19-15:23:47.627410 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00195/att-01/compact.done) = No such file or directory: 11.033 sec on http probe failed: 4, retry 2024/11/19-15:23:48.345660 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-02 2024/11/19-15:23:48.345774 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00194/att-02/compact.done) = No such file or directory: 5.004 sec on http probe failed: 1, retry 2024/11/19-15:23:48.594973 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-00 2024/11/19-15:23:48.595071 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00200/att-00/compact.done) = No such file or directory: 9.018 sec on http probe failed: 3, retry 2024/11/19-15:23:49.550225 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-02 2024/11/19-15:23:49.550329 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00196/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:23:49.632162 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-01 2024/11/19-15:23:49.632241 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00195/att-01/compact.done) = No such file or directory: 13.040 sec on http probe failed: 5, giveup 2024/11/19-15:23:49.632260 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00195/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.640 sec, real 13.311 (submit 265.718 ms wait 13.040 s), input 255.640 MiB, send shutdown 2024/11/19-15:23:49.639983 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-01 2024/11/19-15:23:49.640326 139851148035840 [dcompact_etcd.cc:1773] job-00195/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:49.650202 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005010.sst): old prealloc_size = 147639500, real_size = 103796528 2024/11/19-15:23:49.650319 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005010.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 200576 prefix = 4 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 raw-val = 0.1294 GB zip-val = 0.1034 GB avg-val = 645.16 avg-zval = 515.55 2024/11/19-15:23:49.650363 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005010.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0005 GB, waited 2.240 sec, Key+Value = 0.134 GB 2024/11/19-15:23:49.913681 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005010.sst): second pass time = 2.24's, 57.783'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.26's, 398.921'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3243.409'MB/sec (index lex order gen) rebuild zvType time = 0.22's, 0.227'MB/sec write SST data time = 0.04's, 2493.335'MB/sec dict compress time = 0.00's, 1043.302'MB/sec z-dict build time = 0.00's, sample length = 1.350'MB, throughput = 14633.748'MB/sec zip my value time = 2.24's, unzip length = 0.129'GB zip my value throughput = 57.783'MB/sec zip pipeline throughput = 57.783'MB/sec entries = 200576 avg-key = 24.00 avg-zkey = 1.69 avg-val = 645.16 avg-zval = 515.55 usrkeys = 200576 avg-key = 16.00 avg-zkey = 1.69 avg-val = 645.16 avg-zval = 515.55 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0032 GB value = 0.1294 GB dict = 1.35 MB all = 0.1342 GB } __ZipSize{ index = 0.0003 GB value = 0.1034 GB dict = 1.35 MB all = 0.1051 GB } UnZip/Zip{ index = 9.4790 value = 1.2514 dict = 1.00 all = 1.2771 } Zip/UnZip{ index = 0.1055 value = 0.7991 dict = 1.00 all = 0.7830 } ---------------------------- total value len = 11.273346 GB avg = 0.259 KB (by entry num) total key len = 1.117077 GB avg = 0.026 KB total ukey len = 0.768842 GB avg = 0.018 KB total ukey num = 0.043529440 Billion total entry num = 0.043529440 Billion write speed all = 39.592911207 MB/sec (with seq num) write speed all = 38.480143933 MB/sec (without seq num) 2024/11/19-15:23:49.916182 139851127064320 [compaction_job.cc:1948] [default] [JOB 189] Generated table #5010: 200576 keys, 105099111 bytes, temperature: kUnknown 2024/11/19-15:23:49.916234 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001029916209, "cf_name": "default", "job": 189, "event": "table_file_creation", "file_number": 5010, "file_size": 105099111, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 103407648, "index_size": 338560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4813824, "raw_average_key_size": 24, "raw_value_size": 129403958, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 200576, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001027, "oldest_key_time": 0, "file_creation_time": 1732001027, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5010, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:49.917420 139851148035840 [dcompact_etcd.cc:1163] job-00195/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.277 sec, input = 255.640 MiB 2024/11/19-15:23:49.941529 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005011.sst) /storage/mytopling/..//tmp/Topling-82038-WHr18Q 2024/11/19-15:23:50.065798 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-01 2024/11/19-15:23:50.065908 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00198/att-01/compact.done) = No such file or directory: 4.613 sec on http probe failed: 1, retry 2024/11/19-15:23:50.071752 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005011.sst): first pass time = 0.13's,1018.537'MB/sec 2024/11/19-15:23:50.071757 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005011.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0018 GB, waited 0.000 sec, Key+Value = 0.133 GB 2024/11/19-15:23:50.071780 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005011.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.002 GB, dictZip workingMem = 0.0080 GB, waited 0.000 sec, Key+Value = 0.133 GB 2024/11/19-15:23:50.081953 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005011.sst): index pass time = 0.01's, 233.860'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 198196 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 2024/11/19-15:23:50.357731 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-02 2024/11/19-15:23:50.357835 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00194/att-02/compact.done) = No such file or directory: 7.024 sec on http probe failed: 2, retry 2024/11/19-15:23:50.612185 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-00 2024/11/19-15:23:50.612306 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00200/att-00/compact.done) = No such file or directory: 11.028 sec on http probe failed: 4, retry 2024/11/19-15:23:51.566008 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-02 2024/11/19-15:23:51.566108 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00196/att-02/compact.done) = No such file or directory: 5.186 sec on http probe failed: 2, retry 2024/11/19-15:23:51.804987 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-01 2024/11/19-15:23:51.805096 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00199/att-01/compact.done) = No such file or directory: 5.002 sec on http probe failed: 1, retry 2024/11/19-15:23:52.079170 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-01 2024/11/19-15:23:52.079280 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00198/att-01/compact.done) = No such file or directory: 6.623 sec on http probe failed: 2, retry 2024/11/19-15:23:52.212020 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005011.sst): old prealloc_size = 147639500, real_size = 102563472 2024/11/19-15:23:52.212147 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005011.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 198196 prefix = 4 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 raw-val = 0.1279 GB zip-val = 0.1022 GB avg-val = 645.15 avg-zval = 515.55 2024/11/19-15:23:52.212201 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005011.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 2.140 sec, Key+Value = 0.133 GB 2024/11/19-15:23:52.366721 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-02 2024/11/19-15:23:52.366826 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00194/att-02/compact.done) = No such file or directory: 9.036 sec on http probe failed: 3, retry 2024/11/19-15:23:52.411556 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005011.sst): second pass time = 2.14's, 59.744'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.20's, 520.520'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3073.651'MB/sec (index lex order gen) rebuild zvType time = 0.16's, 0.317'MB/sec write SST data time = 0.04's, 2407.231'MB/sec dict compress time = 0.00's, 1894.120'MB/sec z-dict build time = 0.00's, sample length = 1.339'MB, throughput = 24512.004'MB/sec zip my value time = 2.14's, unzip length = 0.128'GB zip my value throughput = 59.744'MB/sec zip pipeline throughput = 59.744'MB/sec entries = 198196 avg-key = 24.00 avg-zkey = 1.69 avg-val = 645.15 avg-zval = 515.55 usrkeys = 198196 avg-key = 16.00 avg-zkey = 1.69 avg-val = 645.15 avg-zval = 515.55 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0032 GB value = 0.1279 GB dict = 1.34 MB all = 0.1326 GB } __ZipSize{ index = 0.0003 GB value = 0.1022 GB dict = 1.34 MB all = 0.1039 GB } UnZip/Zip{ index = 9.4758 value = 1.2514 dict = 1.00 all = 1.2770 } Zip/UnZip{ index = 0.1055 value = 0.7991 dict = 1.00 all = 0.7831 } ---------------------------- total value len = 11.401212 GB avg = 0.261 KB (by entry num) total key len = 1.121834 GB avg = 0.026 KB total ukey len = 0.772013 GB avg = 0.018 KB total ukey num = 0.043727636 Billion total entry num = 0.043727636 Billion write speed all = 39.699820114 MB/sec (with seq num) write speed all = 38.590837952 MB/sec (without seq num) 2024/11/19-15:23:52.413956 139851127064320 [compaction_job.cc:1948] [default] [JOB 189] Generated table #5011: 198196 keys, 103855128 bytes, temperature: kUnknown 2024/11/19-15:23:52.414005 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032413983, "cf_name": "default", "job": 189, "event": "table_file_creation", "file_number": 5011, "file_size": 103855128, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 102179072, "index_size": 334656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4756704, "raw_average_key_size": 24, "raw_value_size": 127865254, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 198196, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001029, "oldest_key_time": 0, "file_creation_time": 1732001029, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5011, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:52.441542 139851127064320 [compaction_job.cc:703] job-00189: subcompact[0], size: 0.208954 G, files: /storage/mytopling/.rocksdb [005010,005011] 2024/11/19-15:23:52.443411 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 16777215 2024/11/19-15:23:52.443590 139851127064320 (Original Log Time 2024/11/19-15:23:47.253105) EVENT_LOG_v1 {"time_micros": 1732001027253089, "job": 189, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4953, 4954, 4956, 4959], "score": 2.51838, "input_data_size": 259250868, "oldest_snapshot_seqno": 138767053} 2024/11/19-15:23:52.443598 139851127064320 (Original Log Time 2024/11/19-15:23:52.443086) [compaction_job.cc:2022] [default] [JOB 189] Compacted 4@2 files to L3 => 208954239 bytes 2024/11/19-15:23:52.443601 139851127064320 (Original Log Time 2024/11/19-15:23:52.443377) [compaction_job.cc:1224] [default] compacted to: files[7 39 71 42 0 0 0] max score 2.92 (94 files need compaction), MB/sec: 50.0 rd, 40.3 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(247.2, 0.0 +0.0 blob) out(199.3 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 398772, records dropped: 0 output_compression: Snappy 2024/11/19-15:23:52.443606 139851127064320 (Original Log Time 2024/11/19-15:23:52.443404) EVENT_LOG_v1 {"time_micros": 1732001032443391, "cf": "default", "job": 189, "event": "compaction_finished", "compaction_time_micros": 5188455, "compaction_time_cpu_micros": 1570758, "output_level": 3, "num_output_files": 2, "total_output_size": 208954239, "num_input_records": 398772, "num_output_records": 398772, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [7, 39, 71, 42, 0, 0, 0]} 2024/11/19-15:23:52.448119 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004851.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.448146 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032448143, "job": 189, "event": "table_file_deletion", "file_number": 4851} 2024/11/19-15:23:52.452348 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004852.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.452369 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032452367, "job": 189, "event": "table_file_deletion", "file_number": 4852} 2024/11/19-15:23:52.456758 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004854.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.456779 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032456776, "job": 189, "event": "table_file_deletion", "file_number": 4854} 2024/11/19-15:23:52.461112 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004855.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.461133 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032461131, "job": 189, "event": "table_file_deletion", "file_number": 4855} 2024/11/19-15:23:52.465513 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004856.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.465531 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032465529, "job": 189, "event": "table_file_deletion", "file_number": 4856} 2024/11/19-15:23:52.469883 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004858.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.469902 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032469900, "job": 189, "event": "table_file_deletion", "file_number": 4858} 2024/11/19-15:23:52.474273 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004859.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.474292 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032474290, "job": 189, "event": "table_file_deletion", "file_number": 4859} 2024/11/19-15:23:52.478664 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004860.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.478682 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032478680, "job": 189, "event": "table_file_deletion", "file_number": 4860} 2024/11/19-15:23:52.482998 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004862.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.483016 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032483014, "job": 189, "event": "table_file_deletion", "file_number": 4862} 2024/11/19-15:23:52.487342 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004863.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.487360 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032487358, "job": 189, "event": "table_file_deletion", "file_number": 4863} 2024/11/19-15:23:52.491753 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004864.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.491770 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032491768, "job": 189, "event": "table_file_deletion", "file_number": 4864} 2024/11/19-15:23:52.496300 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004866.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.496319 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032496317, "job": 189, "event": "table_file_deletion", "file_number": 4866} 2024/11/19-15:23:52.500704 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004867.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.500722 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032500720, "job": 189, "event": "table_file_deletion", "file_number": 4867} 2024/11/19-15:23:52.504989 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004868.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.505007 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032505005, "job": 189, "event": "table_file_deletion", "file_number": 4868} 2024/11/19-15:23:52.509270 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004870.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.509288 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032509286, "job": 189, "event": "table_file_deletion", "file_number": 4870} 2024/11/19-15:23:52.513338 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004871.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.513357 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032513355, "job": 189, "event": "table_file_deletion", "file_number": 4871} 2024/11/19-15:23:52.513618 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004963.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:23:52.513629 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001032513628, "job": 189, "event": "table_file_deletion", "file_number": 4963} 2024/11/19-15:23:52.513790 139851127064320 [compaction_job.cc:2437] [default] [JOB 201] Compacting 1@2 files to L3, score 1.98, subcompactions 1 : 1 2024/11/19-15:23:52.513798 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 199 Base level 2, inputs: [4621(61MB)] 2024/11/19-15:23:52.617452 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-00 2024/11/19-15:23:52.617525 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00200/att-00/compact.done) = No such file or directory: 13.045 sec on http probe failed: 5, giveup 2024/11/19-15:23:52.617543 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00200/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.783 sec, real 13.340 (submit 290.031 ms wait 13.045 s), input 253.783 MiB, send shutdown 2024/11/19-15:23:52.623134 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-00 2024/11/19-15:23:52.623329 139851169007360 [dcompact_etcd.cc:1773] job-00200/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:52.806204 139851127064320 [dcompact_etcd.cc:1163] job-00201/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.292 sec, input = 62.811 MiB 2024/11/19-15:23:52.914208 139851169007360 [dcompact_etcd.cc:1163] job-00200/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.291 sec, input = 253.783 MiB 2024/11/19-15:23:53.574397 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-02 2024/11/19-15:23:53.574481 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00196/att-02/compact.done) = No such file or directory: 7.199 sec on http probe failed: 3, retry 2024/11/19-15:23:53.810482 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-01 2024/11/19-15:23:53.810543 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00199/att-01/compact.done) = No such file or directory: 7.018 sec on http probe failed: 2, retry 2024/11/19-15:23:54.085299 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-01 2024/11/19-15:23:54.085376 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00198/att-01/compact.done) = No such file or directory: 8.637 sec on http probe failed: 3, retry 2024/11/19-15:23:54.374936 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-02 2024/11/19-15:23:54.375030 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00194/att-02/compact.done) = No such file or directory: 11.045 sec on http probe failed: 4, retry 2024/11/19-15:23:54.382402 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-00 2024/11/19-15:23:54.382487 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00201/att-00/compact.done) = No such file or directory: 1.570 sec on http probe failed: 1, retry 2024/11/19-15:23:54.923610 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-02 2024/11/19-15:23:54.923698 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00195/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:23:55.583812 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-02 2024/11/19-15:23:55.583911 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00196/att-02/compact.done) = No such file or directory: 9.207 sec on http probe failed: 4, retry 2024/11/19-15:23:55.647300 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-00 2024/11/19-15:23:55.647381 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00201/att-00/compact.done) = No such file or directory: 2.833 sec on http probe failed: 2, retry 2024/11/19-15:23:55.819973 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-01 2024/11/19-15:23:55.820072 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00199/att-01/compact.done) = No such file or directory: 9.023 sec on http probe failed: 3, retry 2024/11/19-15:23:56.094515 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-01 2024/11/19-15:23:56.094623 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00198/att-01/compact.done) = No such file or directory: 10.643 sec on http probe failed: 4, retry 2024/11/19-15:23:56.380605 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-02 2024/11/19-15:23:56.380708 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00194/att-02/compact.done) = No such file or directory: 13.054 sec on http probe failed: 5, giveup 2024/11/19-15:23:56.380727 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00194/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.642 sec, real 13.354 (submit 295.213 ms wait 13.054 s), input 255.642 MiB, send shutdown 2024/11/19-15:23:56.388303 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00194/att-02 2024/11/19-15:23:56.388520 139851179493120 [dcompact_etcd.cc:1773] job-00194/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:56.388756 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00194) = success 2024/11/19-15:23:56.388762 139851179493120 [dcompact_etcd.cc:1773] job-00194/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:56.388780 139851179493120 (Original Log Time 2024/11/19-15:23:16.329899) EVENT_LOG_v1 {"time_micros": 1732000996329891, "job": 194, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4968, 4969, 4970, 4971], "score": 2.60155, "input_data_size": 221704736, "oldest_snapshot_seqno": -1} 2024/11/19-15:23:56.388787 139851179493120 [compaction_job.cc:2437] [default] [JOB 194] Compacting 4@2 files to L3, score 2.60, subcompactions 1 : 1 2024/11/19-15:23:56.388804 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 192 Base level 2, inputs: [4968(52MB) 4969(52MB) 4970(52MB) 4971(52MB)] 2024/11/19-15:23:56.389060 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005012.sst) /storage/mytopling/..//tmp/Topling-82038-EZca8u 2024/11/19-15:23:56.713093 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005012.sst): first pass time = 0.32's, 414.230'MB/sec 2024/11/19-15:23:56.713095 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005012.sst: sumWaitingMem = 0.012 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0120 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:23:56.713134 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005012.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.012 GB, dictZip workingMem = 0.0065 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:23:56.798223 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005012.sst): index pass time = 0.09's, 174.288'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 926867 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:23:56.912638 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-00 2024/11/19-15:23:56.912749 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00201/att-00/compact.done) = No such file or directory: 4.097 sec on http probe failed: 3, retry 2024/11/19-15:23:56.931608 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-02 2024/11/19-15:23:56.931681 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00195/att-02/compact.done) = No such file or directory: 7.006 sec on http probe failed: 2, retry 2024/11/19-15:23:57.600645 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-02 2024/11/19-15:23:57.600761 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00196/att-02/compact.done) = No such file or directory: 11.221 sec on http probe failed: 5, giveup 2024/11/19-15:23:57.600796 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00196/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.899 sec, real 11.500 (submit 266.649 ms wait 11.221 s), input 126.899 MiB, send shutdown 2024/11/19-15:23:57.610125 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00196/att-02 2024/11/19-15:23:57.610418 139851137550080 [dcompact_etcd.cc:1773] job-00196/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:57.610694 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00196) = success 2024/11/19-15:23:57.610705 139851137550080 [dcompact_etcd.cc:1773] job-00196/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:57.610731 139851137550080 (Original Log Time 2024/11/19-15:23:23.069843) EVENT_LOG_v1 {"time_micros": 1732001003069837, "job": 196, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4885, 4887, 4888, 4889], "score": 2.42301, "input_data_size": 134099776, "oldest_snapshot_seqno": 139670752} 2024/11/19-15:23:57.610741 139851137550080 [compaction_job.cc:2437] [default] [JOB 196] Compacting 4@1 files to L2, score 2.42, subcompactions 1 : 1 2024/11/19-15:23:57.610753 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 194 Base level 1, inputs: [4885(31MB) 4887(31MB) 4888(31MB) 4889(31MB)] 2024/11/19-15:23:57.611039 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005013.sst) /storage/mytopling/..//tmp/Topling-82038-iwDRv8 2024/11/19-15:23:57.774405 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005013.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.006 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:23:57.779748 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005013.sst): index pass time = 0.01's, 97.617'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64713 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:23:57.828982 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-01 2024/11/19-15:23:57.829088 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00199/att-01/compact.done) = No such file or directory: 11.037 sec on http probe failed: 4, retry 2024/11/19-15:23:57.902681 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005013.sst): first pass time = 0.29's, 230.124'MB/sec 2024/11/19-15:23:57.902751 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005013.sst): old prealloc_size = 73819750, real_size = 66152224 2024/11/19-15:23:57.902832 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005013.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64713 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.03 avg-zval =1020.28 2024/11/19-15:23:57.902897 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005013.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0001 GB, waited 0.128 sec, Key+Value = 0.067 GB 2024/11/19-15:23:57.930323 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-01 2024/11/19-15:23:57.930462 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00200/att-01/compact.done) = No such file or directory: 5.001 sec on http probe failed: 1, retry 2024/11/19-15:23:57.983950 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005013.sst): second pass time = 0.00's,216497203.947'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.08's, 814.669'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1753.564'MB/sec (index lex order gen) rebuild zvType time = 0.06's, 0.269'MB/sec write SST data time = 0.02's, 3140.260'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 216497203.947'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64713 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 1020.28 usrkeys = 64713 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 1020.28 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0463 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 11.467027 GB avg = 0.262 KB (by entry num) total key len = 1.123128 GB avg = 0.026 KB total ukey len = 0.772790 GB avg = 0.018 KB total ukey num = 0.043792349 Billion total entry num = 0.043792349 Billion write speed all = 39.219740033 MB/sec (with seq num) write speed all = 38.128395545 MB/sec (without seq num) 2024/11/19-15:23:57.987834 139851137550080 [compaction_job.cc:1948] [default] [JOB 196] Generated table #5013: 64713 keys, 66137476 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:57.987885 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001037987862, "cf_name": "default", "job": 196, "event": "table_file_creation", "file_number": 5013, "file_size": 66137476, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025680, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294260, "raw_average_key_size": 20, "raw_value_size": 65815150, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64713, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001037, "oldest_key_time": 0, "file_creation_time": 1732001037, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5013, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:58.002843 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005014.sst) /storage/mytopling/..//tmp/Topling-82038-XofFAY 2024/11/19-15:23:58.056727 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005014.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.006 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:23:58.056827 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005014.sst): index pass time = 0.00's,9085.390'MB/sec index type = UintIndex_AllOne usrkeys = 63600 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:23:58.103576 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-01 2024/11/19-15:23:58.103691 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00198/att-01/compact.done) = No such file or directory: 12.652 sec on http probe failed: 5, giveup 2024/11/19-15:23:58.103716 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00198/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 184.525 sec, real 12.932 (submit 270.893 ms wait 12.652 s), input 184.525 MiB, send shutdown 2024/11/19-15:23:58.109986 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-01 2024/11/19-15:23:58.110229 139850978166528 [dcompact_etcd.cc:1773] job-00198/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:58.123995 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005014.sst): first pass time = 0.12's, 544.464'MB/sec 2024/11/19-15:23:58.124039 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005014.sst): old prealloc_size = 73819750, real_size = 64904872 2024/11/19-15:23:58.124062 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005014.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63600 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:23:58.176771 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-00 2024/11/19-15:23:58.176914 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00201/att-00/compact.done) = No such file or directory: 5.363 sec on http probe failed: 4, retry 2024/11/19-15:23:58.189635 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005014.sst): second pass time = 0.00's,223040379.310'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 989.426'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.853'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.361'MB/sec write SST data time = 0.02's, 3019.350'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 223040379.310'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63600 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 63600 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0000 GB value = 0.0649 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index =3975.0000 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 11.531708 GB avg = 0.263 KB (by entry num) total key len = 1.124400 GB avg = 0.026 KB total ukey len = 0.773553 GB avg = 0.018 KB total ukey num = 0.043855949 Billion total entry num = 0.043855949 Billion write speed all = 39.399946870 MB/sec (with seq num) write speed all = 38.307717282 MB/sec (without seq num) 2024/11/19-15:23:58.206553 139851137550080 [compaction_job.cc:1948] [default] [JOB 196] Generated table #5014: 63600 keys, 64889930 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.206606 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001038206584, "cf_name": "default", "job": 196, "event": "table_file_creation", "file_number": 5014, "file_size": 64889930, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64888624, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1272000, "raw_average_key_size": 20, "raw_value_size": 64681710, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63600, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5014, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:58.229448 139851137550080 [compaction_job.cc:703] job-00196: subcompact[0], size: 0.131027 G, files: /storage/mytopling/.rocksdb [005013,005014] 2024/11/19-15:23:58.230364 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 16777216 2024/11/19-15:23:58.230516 139851137550080 (Original Log Time 2024/11/19-15:23:57.610798) EVENT_LOG_v1 {"time_micros": 1732001037610778, "job": 196, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [4885, 4887, 4888, 4889], "score": 2.42301, "input_data_size": 134099776, "oldest_snapshot_seqno": 139670752} 2024/11/19-15:23:58.230525 139851137550080 (Original Log Time 2024/11/19-15:23:58.229956) [compaction_job.cc:2022] [default] [JOB 196] Compacted 4@1 files to L2 => 131027406 bytes 2024/11/19-15:23:58.230531 139851137550080 (Original Log Time 2024/11/19-15:23:58.230320) [compaction_job.cc:1224] [default] compacted to: files[7 35 73 42 0 0 0] max score 3.25 (95 files need compaction), MB/sec: 216.8 rd, 211.8 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128313, records dropped: 0 output_compression: Snappy 2024/11/19-15:23:58.230555 139851137550080 (Original Log Time 2024/11/19-15:23:58.230355) EVENT_LOG_v1 {"time_micros": 1732001038230339, "cf": "default", "job": 196, "event": "compaction_finished", "compaction_time_micros": 618668, "compaction_time_cpu_micros": 479734, "output_level": 2, "num_output_files": 2, "total_output_size": 131027406, "num_input_records": 128313, "num_output_records": 128313, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [7, 35, 73, 42, 0, 0, 0]} 2024/11/19-15:23:58.250374 139851137550080 [compaction_job.cc:2437] [default] [JOB 202] Compacting 7@0 + 35@1 files to L1, score 3.25, subcompactions 3 : 3 2024/11/19-15:23:58.250401 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 200 Base level 0, inputs: [5004(514MB) 4985(514MB) 4977(513MB) 4966(520MB) 4961(516MB) 4943(515MB) 4904(521MB)], [4891(31MB) 4892(31MB) 4894(31MB) 4895(31MB) 4897(31MB) 4898(31MB) 4900(31MB) 4901(31MB) 4902(31MB) 4906(31MB) 4907(31MB) 4909(31MB) 4910(31MB) 4911(31MB) 4913(31MB) 4914(31MB) 4915(31MB) 4917(31MB) 4918(31MB) 4919(31MB) 4921(31MB) 4922(31MB) 4923(31MB) 4924(31MB) 4925(31MB) 4926(31MB) 4927(31MB) 4928(31MB) 4929(31MB) 4930(31MB) 4931(31MB) 4932(31MB) 4933(31MB) 4934(31MB) 4935(25MB)] 2024/11/19-15:23:58.266756 139851137550080 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5015: 20 keys, 6963 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.266804 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001038266783, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5015, "file_size": 6963, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130572162, "largest_seqno": 139714809, "table_properties": {"data_size": 2304, "index_size": 3840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 320, "raw_average_key_size": 16, "raw_value_size": 2300, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 20, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5015, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.278437 139851137550080 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5018: 193 keys, 33144 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.278480 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001038278460, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5018, "file_size": 33144, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130203043, "largest_seqno": 139804824, "table_properties": {"data_size": 23424, "index_size": 8896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3860, "raw_average_key_size": 20, "raw_value_size": 23386, "raw_average_value_size": 121, "num_data_blocks": 1, "num_entries": 193, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5018, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.309039 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5017: 32057 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.309137 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001038309103, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5017, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 124884486, "largest_seqno": 124965408, "table_properties": {"data_size": 32601344, "index_size": 922944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601334, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5017, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.339819 139851137550080 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5019: 49859 keys, 33507200 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.339889 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001038339856, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5019, "file_size": 33507200, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130203948, "largest_seqno": 131123844, "table_properties": {"data_size": 32158336, "index_size": 1348032, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196616, "raw_average_key_size": 24, "raw_value_size": 32158308, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49859, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5019, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.368980 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5020: 32055 keys, 33524352 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.369091 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001038369043, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5020, "file_size": 33524352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 124965409, "largest_seqno": 125595969, "table_properties": {"data_size": 32600768, "index_size": 922752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32600718, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5020, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.369399 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005012.sst): old prealloc_size = 147639500, real_size = 36348704 2024/11/19-15:23:58.370065 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005012.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926867 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.81 avg-zval = 37.06 2024/11/19-15:23:58.370151 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005012.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 1.657 sec, Key+Value = 0.134 GB 2024/11/19-15:23:58.418592 139851137550080 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5021: 49867 keys, 33506560 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.418662 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001038418630, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5021, "file_size": 33506560, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130965477, "largest_seqno": 131900829, "table_properties": {"data_size": 32156800, "index_size": 1348928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196808, "raw_average_key_size": 24, "raw_value_size": 32156791, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49867, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5021, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.429207 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5016: 291061 keys, 22885443 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.429267 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001038429246, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5016, "file_size": 22885443, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134959886, "largest_seqno": 139660751, "table_properties": {"data_size": 15717568, "index_size": 7167040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 5821220, "raw_average_key_size": 20, "raw_value_size": 15717518, "raw_average_value_size": 54, "num_data_blocks": 1, "num_entries": 291061, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5016, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.438698 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5022: 32129 keys, 33524288 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.438744 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001038438723, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5022, "file_size": 33524288, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125595970, "largest_seqno": 125698905, "table_properties": {"data_size": 32676096, "index_size": 847360, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642580, "raw_average_key_size": 20, "raw_value_size": 32676041, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32129, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5022, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.448476 139850978166528 [dcompact_etcd.cc:1163] job-00198/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.338 sec, input = 184.525 MiB 2024/11/19-15:23:58.477042 139851137550080 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5023: 49874 keys, 33506752 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.477092 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001038477070, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5023, "file_size": 33506752, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 131900830, "largest_seqno": 132523383, "table_properties": {"data_size": 32158464, "index_size": 1347456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196976, "raw_average_key_size": 24, "raw_value_size": 32158425, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49874, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5023, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.477131 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5025: 32059 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.477174 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001038477153, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5025, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125698906, "largest_seqno": 125780990, "table_properties": {"data_size": 32602112, "index_size": 921920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32602066, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5025, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.539762 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5024: 174600 keys, 4543502 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.539816 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001038539794, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5024, "file_size": 4543502, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130202142, "largest_seqno": 139714807, "table_properties": {"data_size": 174656, "index_size": 4368000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4190400, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 174600, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5024, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.539869 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5027: 32052 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.539913 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001038539894, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5027, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125761863, "largest_seqno": 125821870, "table_properties": {"data_size": 32597056, "index_size": 926976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641040, "raw_average_key_size": 20, "raw_value_size": 32597008, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32052, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5027, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.540108 139851137550080 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5026: 49851 keys, 33507136 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.540151 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001038540130, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5026, "file_size": 33507136, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 132523384, "largest_seqno": 133518342, "table_properties": {"data_size": 32158592, "index_size": 1347712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196424, "raw_average_key_size": 24, "raw_value_size": 32158534, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49851, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5026, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.598870 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5029: 32151 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.598930 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001038598901, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5029, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125783044, "largest_seqno": 125844207, "table_properties": {"data_size": 32697088, "index_size": 827264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32697057, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5029, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.609791 139851137550080 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5030: 49815 keys, 33505664 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.609857 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001038609825, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5030, "file_size": 33505664, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 133283905, "largest_seqno": 134517354, "table_properties": {"data_size": 32155200, "index_size": 1349632, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1195560, "raw_average_key_size": 24, "raw_value_size": 32155183, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49815, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5030, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.712335 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5031: 32053 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.712444 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001038712396, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5031, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125844208, "largest_seqno": 125953055, "table_properties": {"data_size": 32598080, "index_size": 925824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32598032, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5031, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.813849 139851137550080 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5032: 49859 keys, 33506816 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.813908 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001038813885, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5032, "file_size": 33506816, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134044444, "largest_seqno": 135354951, "table_properties": {"data_size": 32157888, "index_size": 1348096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196616, "raw_average_key_size": 24, "raw_value_size": 32157857, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49859, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5032, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.813953 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5033: 32055 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.814013 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001038813984, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5033, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 125953056, "largest_seqno": 126577988, "table_properties": {"data_size": 32599680, "index_size": 923968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32599656, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5033, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.867223 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5035: 32064 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.867320 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001038867278, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5035, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126577989, "largest_seqno": 126659194, "table_properties": {"data_size": 32611008, "index_size": 912896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641280, "raw_average_key_size": 20, "raw_value_size": 32610954, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32064, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5035, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.870450 139851137550080 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5034: 49851 keys, 33507072 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.870537 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001038870492, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5034, "file_size": 33507072, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134959948, "largest_seqno": 136112252, "table_properties": {"data_size": 32158656, "index_size": 1347584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196424, "raw_average_key_size": 24, "raw_value_size": 32158593, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49851, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5034, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.871494 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5028: 576000 keys, 25500931 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.871572 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001038871529, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5028, "file_size": 25500931, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130215952, "largest_seqno": 139673753, "table_properties": {"data_size": 11404800, "index_size": 14095296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13824000, "raw_average_key_size": 24, "raw_value_size": 11404800, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 576000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5028, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.927942 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5036: 32054 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.927998 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001038927976, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5036, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126659195, "largest_seqno": 126762085, "table_properties": {"data_size": 32598912, "index_size": 925440, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641080, "raw_average_key_size": 20, "raw_value_size": 32598870, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5036, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.939154 139851137550080 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5037: 49843 keys, 33506816 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.939221 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001038939190, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5037, "file_size": 33506816, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 135957863, "largest_seqno": 136823081, "table_properties": {"data_size": 32159040, "index_size": 1346944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196232, "raw_average_key_size": 24, "raw_value_size": 32159022, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49843, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5037, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.941348 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-02 2024/11/19-15:23:58.941485 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00195/att-02/compact.done) = No such file or directory: 9.014 sec on http probe failed: 3, retry 2024/11/19-15:23:58.977292 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5039: 32055 keys, 33524352 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.977376 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001038977340, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5039, "file_size": 33524352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126762086, "largest_seqno": 126810536, "table_properties": {"data_size": 32600192, "index_size": 923328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32600176, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5039, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:58.987617 139851137550080 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5040: 49874 keys, 33506944 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:58.987679 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001038987651, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5040, "file_size": 33506944, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 136823082, "largest_seqno": 137519613, "table_properties": {"data_size": 32158144, "index_size": 1347968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196976, "raw_average_key_size": 24, "raw_value_size": 32158115, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49874, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5040, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.040274 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5041: 32076 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.040365 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039040329, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5041, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126774143, "largest_seqno": 126832230, "table_properties": {"data_size": 32624000, "index_size": 900160, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641520, "raw_average_key_size": 20, "raw_value_size": 32623984, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32076, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5041, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.058484 139851137550080 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5042: 49876 keys, 33506880 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.058590 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001039058545, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5042, "file_size": 33506880, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137519614, "largest_seqno": 138510114, "table_properties": {"data_size": 32156160, "index_size": 1349888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1197024, "raw_average_key_size": 24, "raw_value_size": 32156101, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49876, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5042, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.077959 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005012.sst): second pass time = 1.66's, 65.369'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.71's, 52.514'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2697.467'MB/sec (index lex order gen) rebuild zvType time = 0.51's, 0.453'MB/sec write SST data time = 0.20's, 189.883'MB/sec dict compress time = 0.00's, 1414.449'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18302.831'MB/sec zip my value time = 1.66's, unzip length = 0.108'GB zip my value throughput = 65.369'MB/sec zip pipeline throughput = 65.369'MB/sec entries = 926867 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.81 avg-zval = 37.06 usrkeys = 926867 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.81 avg-zval = 37.06 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 10 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5116 value = 3.1515 dict = 1.00 all = 3.6074 } Zip/UnZip{ index = 0.0951 value = 0.3173 dict = 1.00 all = 0.2772 } ---------------------------- total value len = 11.639974 GB avg = 0.260 KB (by entry num) total key len = 1.150353 GB avg = 0.026 KB total ukey len = 0.792090 GB avg = 0.018 KB total ukey num = 0.044782816 Billion total entry num = 0.044782816 Billion write speed all = 39.707979044 MB/sec (with seq num) write speed all = 38.595741543 MB/sec (without seq num) 2024/11/19-15:23:59.079847 139851179493120 [compaction_job.cc:1948] [default] [JOB 194] Generated table #5012: 926867 keys, 37206689 bytes, temperature: kUnknown 2024/11/19-15:23:59.079958 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001039079894, "cf_name": "default", "job": 194, "event": "table_file_creation", "file_number": 5012, "file_size": 37206689, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34353296, "index_size": 1763520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25952276, "raw_average_key_size": 28, "raw_value_size": 108265586, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926867, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001036, "oldest_key_time": 0, "file_creation_time": 1732001036, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5012, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:23:59.096891 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5043: 32054 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.096983 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039096947, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5043, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126832231, "largest_seqno": 126907321, "table_properties": {"data_size": 32599296, "index_size": 924288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641080, "raw_average_key_size": 20, "raw_value_size": 32599261, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5043, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.098293 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5038: 231234 keys, 33325251 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.098366 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001039098330, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5038, "file_size": 33325251, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130221954, "largest_seqno": 130571259, "table_properties": {"data_size": 27032704, "index_size": 6291712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6474552, "raw_average_key_size": 28, "raw_value_size": 27032704, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231234, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001038, "oldest_key_time": 0, "file_creation_time": 1732001038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5038, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.117217 139851137550080 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5044: 49869 keys, 33507136 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.117306 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001039117264, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5044, "file_size": 33507136, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138271341, "largest_seqno": 139516882, "table_properties": {"data_size": 32158336, "index_size": 1347968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196856, "raw_average_key_size": 24, "raw_value_size": 32158292, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49869, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5044, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.118359 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005048.sst) /storage/mytopling/..//tmp/Topling-82038-BZPrvQ 2024/11/19-15:23:59.146963 139851137550080 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5047: 27562 keys, 18604416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.147034 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001039147006, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5047, "file_size": 18604416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138918180, "largest_seqno": 139657750, "table_properties": {"data_size": 17796160, "index_size": 807424, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 661488, "raw_average_key_size": 24, "raw_value_size": 17796157, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 27562, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5047, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.157945 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5045: 32152 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.158020 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039157985, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5045, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 126907322, "largest_seqno": 127013241, "table_properties": {"data_size": 32697664, "index_size": 826624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643040, "raw_average_key_size": 20, "raw_value_size": 32697661, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32152, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5045, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.217789 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5050: 32058 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.217842 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039217821, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5050, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127013242, "largest_seqno": 127648857, "table_properties": {"data_size": 32603712, "index_size": 919872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603680, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5050, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.249080 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5051: 32151 keys, 33524224 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.249133 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039249112, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5051, "file_size": 33524224, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127648858, "largest_seqno": 127751612, "table_properties": {"data_size": 32696768, "index_size": 826624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32696746, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5051, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.275960 139851137550080 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5049: 284939 keys, 22410691 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.276016 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001039275993, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5049, "file_size": 22410691, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130209950, "largest_seqno": 134959885, "table_properties": {"data_size": 15390016, "index_size": 7019840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 5698780, "raw_average_key_size": 20, "raw_value_size": 15389990, "raw_average_value_size": 54, "num_data_blocks": 1, "num_entries": 284939, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5049, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.280437 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5052: 32059 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.280480 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039280461, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5052, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127751613, "largest_seqno": 127813681, "table_properties": {"data_size": 32603072, "index_size": 921344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603039, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5052, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.311806 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5046: 231456 keys, 33324931 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.311888 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001039311856, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5046, "file_size": 33324931, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130433325, "largest_seqno": 131073071, "table_properties": {"data_size": 27026560, "index_size": 6297536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480768, "raw_average_key_size": 28, "raw_value_size": 27026536, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231456, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5046, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.315755 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5053: 32058 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.315812 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039315783, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5053, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127757513, "largest_seqno": 127817521, "table_properties": {"data_size": 32604160, "index_size": 919872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32604125, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5053, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.350915 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5055: 32104 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.350971 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039350950, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5055, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127805737, "largest_seqno": 127893930, "table_properties": {"data_size": 32652480, "index_size": 871744, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642080, "raw_average_key_size": 20, "raw_value_size": 32652463, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32104, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5055, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.382179 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5056: 32061 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.382232 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039382211, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5056, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127893931, "largest_seqno": 127968822, "table_properties": {"data_size": 32603264, "index_size": 921216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641220, "raw_average_key_size": 20, "raw_value_size": 32603228, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32061, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5056, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.416332 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #5058. Immutable memtables: 0. 2024/11/19-15:23:59.416654 139852362282752 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 16273899 2024/11/19-15:23:59.417116 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 203] Syncing log #5003 2024/11/19-15:23:59.417217 139850957195008 (Original Log Time 2024/11/19-15:23:59.416796) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:23:59.417225 139850957195008 [flush_job.cc:879] [default] [JOB 203] Flushing memtable with next log file: 5058 2024/11/19-15:23:59.417246 139850957195008 EVENT_LOG_v1 {"time_micros": 1732001039417240, "job": 203, "event": "flush_started", "num_memtables": 1, "num_entries": 1529507, "num_deletes": 0, "total_data_size": 479968100, "memory_usage": 537656712, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:23:59.417250 139850957195008 [flush_job.cc:910] [default] [JOB 203] Level-0 flush table #5059: started 2024/11/19-15:23:59.417706 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000025.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/005059.sst in ConvertToSST, mem_size = 512.749 M, time = 446.518 us 2024/11/19-15:23:59.424334 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5057: 32058 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.424397 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039424369, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5057, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 127968823, "largest_seqno": 128639825, "table_properties": {"data_size": 32603392, "index_size": 920768, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603376, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5057, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.434436 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/005059.sst): fsize = 512.751 M, time(ms): open: 0.042, seek: 0.022, finish: 0.037, meta: 0.012, Flush: 0.011, sync: 16.583, close: 0.008, all: 16.715 2024/11/19-15:23:59.435504 139850957195008 [flush_job.cc:1083] [default] [JOB 203] Flush lasted 18287 microseconds, and 15067 cpu microseconds. 2024/11/19-15:23:59.442771 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-00 2024/11/19-15:23:59.442870 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00201/att-00/compact.done) = No such file or directory: 6.627 sec on http probe failed: 5, giveup 2024/11/19-15:23:59.442886 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00201/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 62.811 sec, real 6.929 (submit 291.974 ms wait 6.627 s), input 62.811 MiB, send shutdown 2024/11/19-15:23:59.456770 139850957195008 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 8 level-0 files rate 9764339 2024/11/19-15:23:59.456842 139850957195008 (Original Log Time 2024/11/19-15:23:59.434474) [flush_job.cc:1029] [default] [JOB 203] Level-0 flush table #5059: 537658138 bytes OK (needs compaction) 2024/11/19-15:23:59.456850 139850957195008 (Original Log Time 2024/11/19-15:23:59.435519) [memtable_list.cc:560] [default] Level-0 commit table #5059 started 2024/11/19-15:23:59.456854 139850957195008 (Original Log Time 2024/11/19-15:23:59.456741) [memtable_list.cc:759] [default] Level-0 commit table #5059: memtable #1 done 2024/11/19-15:23:59.456858 139850957195008 (Original Log Time 2024/11/19-15:23:59.456764) EVENT_LOG_v1 {"time_micros": 1732001039456759, "job": 203, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [8, 35, 73, 42, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:23:59.456862 139850957195008 (Original Log Time 2024/11/19-15:23:59.456814) [db_impl_compaction_flush.cc:310] [default] Level summary: files[8 35 73 42 0 0 0] max score 2.03 (54 files need compaction) 2024/11/19-15:23:59.460576 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5060: 32150 keys, 33524224 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.460619 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039460598, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5060, "file_size": 33524224, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128639826, "largest_seqno": 128741886, "table_properties": {"data_size": 32695872, "index_size": 827520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32695820, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5060, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.464451 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-00 2024/11/19-15:23:59.464659 139851127064320 [dcompact_etcd.cc:1773] job-00201/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:59.491234 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5061: 32056 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.491278 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039491257, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5061, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128741887, "largest_seqno": 128804854, "table_properties": {"data_size": 32602304, "index_size": 921792, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32602264, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5061, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.495118 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5054: 231459 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.495161 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001039495140, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5054, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 131010579, "largest_seqno": 131599799, "table_properties": {"data_size": 27023296, "index_size": 6300864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480852, "raw_average_key_size": 28, "raw_value_size": 27023258, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231459, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5054, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.525902 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5062: 32059 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.525955 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039525935, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5062, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128746200, "largest_seqno": 128826210, "table_properties": {"data_size": 32603648, "index_size": 920384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603625, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5062, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.558670 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5064: 32151 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.558716 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039558696, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5064, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128786912, "largest_seqno": 128848075, "table_properties": {"data_size": 32697152, "index_size": 826560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32697099, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5064, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.592703 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5065: 32060 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.592772 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039592750, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5065, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128848076, "largest_seqno": 128920343, "table_properties": {"data_size": 32602816, "index_size": 921472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32602796, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5065, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.601414 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5063: 231375 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.601480 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001039601451, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5063, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 131094633, "largest_seqno": 131645747, "table_properties": {"data_size": 27029504, "index_size": 6294656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6478500, "raw_average_key_size": 28, "raw_value_size": 27029486, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231375, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5063, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.624352 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5066: 32059 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.624402 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039624381, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5066, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 128920344, "largest_seqno": 129630970, "table_properties": {"data_size": 32601280, "index_size": 922368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32601220, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5066, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.634327 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005048.sst): first pass time = 0.52's, 259.413'MB/sec 2024/11/19-15:23:59.634339 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005048.sst: sumWaitingMem = 0.012 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0120 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:23:59.634371 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005048.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.012 GB, dictZip workingMem = 0.0065 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:23:59.657394 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5068: 32123 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.657445 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039657424, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5068, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 129630971, "largest_seqno": 129700129, "table_properties": {"data_size": 32669504, "index_size": 854528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642460, "raw_average_key_size": 20, "raw_value_size": 32669479, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32123, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5068, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.694399 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5069: 32055 keys, 33524800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.694474 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039694442, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5069, "file_size": 33524800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 129700130, "largest_seqno": 129786203, "table_properties": {"data_size": 32602112, "index_size": 921856, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32602087, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5069, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.717582 139851127064320 [dcompact_etcd.cc:1163] job-00201/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.253 sec, input = 62.811 MiB 2024/11/19-15:23:59.739570 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5070: 32051 keys, 33524352 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.739623 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039739601, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5070, "file_size": 33524352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 129755044, "largest_seqno": 129815052, "table_properties": {"data_size": 32596032, "index_size": 927488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641020, "raw_average_key_size": 20, "raw_value_size": 32596009, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32051, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5070, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.740434 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005048.sst): index pass time = 0.11's, 139.426'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 924222 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0148 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:23:59.756172 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5067: 231326 keys, 33325123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.756225 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001039756202, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5067, "file_size": 33325123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 131645748, "largest_seqno": 132345062, "table_properties": {"data_size": 27029888, "index_size": 6294400, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6477128, "raw_average_key_size": 28, "raw_value_size": 27029856, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231326, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5067, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.771645 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5071: 32150 keys, 33524224 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.771694 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039771673, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5071, "file_size": 33524224, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 129778253, "largest_seqno": 129839415, "table_properties": {"data_size": 32696640, "index_size": 826752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32696625, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5071, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.803660 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5073: 32052 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.803713 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039803692, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5073, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 129839416, "largest_seqno": 129911312, "table_properties": {"data_size": 32597312, "index_size": 926464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641040, "raw_average_key_size": 20, "raw_value_size": 32597296, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32052, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5073, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.835017 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5074: 32056 keys, 33525376 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.835065 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039835045, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5074, "file_size": 33525376, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 129911313, "largest_seqno": 130615610, "table_properties": {"data_size": 32599808, "index_size": 924736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32599767, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5074, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.837664 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-01 2024/11/19-15:23:59.837786 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00199/att-01/compact.done) = No such file or directory: 13.042 sec on http probe failed: 5, giveup 2024/11/19-15:23:59.837812 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00199/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.865 sec, real 13.351 (submit 300.829 ms wait 13.042 s), input 253.865 MiB, send shutdown 2024/11/19-15:23:59.843285 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-01 2024/11/19-15:23:59.843540 139851158521600 [dcompact_etcd.cc:1773] job-00199/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:23:59.862523 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5072: 231478 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.862570 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001039862548, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5072, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 131977830, "largest_seqno": 132566052, "table_properties": {"data_size": 27025600, "index_size": 6298624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481384, "raw_average_key_size": 28, "raw_value_size": 27025548, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231478, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5072, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.874337 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5075: 32151 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.874399 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039874369, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5075, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130615611, "largest_seqno": 130690393, "table_properties": {"data_size": 32697280, "index_size": 826432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32697263, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5075, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.914808 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5077: 32056 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.914858 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039914837, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5077, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130690394, "largest_seqno": 130766545, "table_properties": {"data_size": 32600960, "index_size": 923392, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32600950, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5077, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.939488 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-01 2024/11/19-15:23:59.939609 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00200/att-01/compact.done) = No such file or directory: 7.016 sec on http probe failed: 2, retry 2024/11/19-15:23:59.946139 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5078: 32057 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.946185 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039946164, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5078, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130752875, "largest_seqno": 130812882, "table_properties": {"data_size": 32601536, "index_size": 922944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601476, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5078, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.972252 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5076: 231499 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.972302 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001039972281, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5076, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 132306576, "largest_seqno": 132976907, "table_properties": {"data_size": 27028224, "index_size": 6295936, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481972, "raw_average_key_size": 28, "raw_value_size": 27028198, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231499, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5076, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:23:59.977734 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5079: 32132 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:23:59.977776 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001039977756, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5079, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130758602, "largest_seqno": 130820743, "table_properties": {"data_size": 32680192, "index_size": 843904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642640, "raw_average_key_size": 20, "raw_value_size": 32680161, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32132, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5079, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.009650 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5081: 32058 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:00.009703 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001040009682, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5081, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130820744, "largest_seqno": 130891609, "table_properties": {"data_size": 32602624, "index_size": 921856, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602593, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5081, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.080537 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5082: 32052 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:00.080594 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001040080571, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5082, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130891610, "largest_seqno": 131611676, "table_properties": {"data_size": 32597696, "index_size": 926528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641040, "raw_average_key_size": 20, "raw_value_size": 32597651, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32052, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001040, "oldest_key_time": 0, "file_creation_time": 1732001040, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5082, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.089844 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5080: 231277 keys, 33325251 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:00.089897 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001040089875, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5080, "file_size": 33325251, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 132976908, "largest_seqno": 133556602, "table_properties": {"data_size": 27033600, "index_size": 6290816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6475756, "raw_average_key_size": 28, "raw_value_size": 27033598, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231277, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5080, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.127446 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5083: 32151 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:00.127506 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001040127484, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5083, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 131611677, "largest_seqno": 131714519, "table_properties": {"data_size": 32696704, "index_size": 827072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32696650, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001040, "oldest_key_time": 0, "file_creation_time": 1732001040, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5083, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.159528 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5085: 32051 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:00.159581 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001040159559, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5085, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 131714520, "largest_seqno": 131776579, "table_properties": {"data_size": 32597056, "index_size": 927168, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641020, "raw_average_key_size": 20, "raw_value_size": 32596994, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32051, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001040, "oldest_key_time": 0, "file_creation_time": 1732001040, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5085, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.159612 139851158521600 [dcompact_etcd.cc:1163] job-00199/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.316 sec, input = 253.865 MiB 2024/11/19-15:24:00.196945 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5086: 32150 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:00.197001 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001040196979, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5086, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 131776580, "largest_seqno": 131838740, "table_properties": {"data_size": 32696832, "index_size": 827520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32696824, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001040, "oldest_key_time": 0, "file_creation_time": 1732001040, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5086, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.216866 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5084: 231441 keys, 33324931 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:00.216931 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001040216901, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5084, "file_size": 33324931, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 133006912, "largest_seqno": 133579236, "table_properties": {"data_size": 27027200, "index_size": 6296896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480348, "raw_average_key_size": 28, "raw_value_size": 27027174, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231441, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001040, "oldest_key_time": 0, "file_creation_time": 1732001040, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5084, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.236755 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5087: 32058 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:00.236827 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001040236795, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5087, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 131720401, "largest_seqno": 131840419, "table_properties": {"data_size": 32603392, "index_size": 920448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603348, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001040, "oldest_key_time": 0, "file_creation_time": 1732001040, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5087, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.297102 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5089: 32151 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:00.297177 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001040297146, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5089, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 131780789, "largest_seqno": 131848954, "table_properties": {"data_size": 32696704, "index_size": 827200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32696680, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001040, "oldest_key_time": 0, "file_creation_time": 1732001040, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5089, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.336619 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5088: 231465 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:00.336694 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001040336663, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5088, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 133579237, "largest_seqno": 134218434, "table_properties": {"data_size": 27024320, "index_size": 6299840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481020, "raw_average_key_size": 28, "raw_value_size": 27024266, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231465, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001040, "oldest_key_time": 0, "file_creation_time": 1732001040, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5088, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.360977 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5090: 32056 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:00.361056 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001040361023, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5090, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 131848955, "largest_seqno": 131923694, "table_properties": {"data_size": 32602368, "index_size": 921472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32602363, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001040, "oldest_key_time": 0, "file_creation_time": 1732001040, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5090, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.438976 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5092: 32059 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:00.439050 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001040439018, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5092, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 131923695, "largest_seqno": 132670715, "table_properties": {"data_size": 32604160, "index_size": 920128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32604108, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001040, "oldest_key_time": 0, "file_creation_time": 1732001040, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5092, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.538975 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5093: 32073 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:00.539057 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001040539024, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5093, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 132670716, "largest_seqno": 132746541, "table_properties": {"data_size": 32619264, "index_size": 905024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641460, "raw_average_key_size": 20, "raw_value_size": 32619221, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32073, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001040, "oldest_key_time": 0, "file_creation_time": 1732001040, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5093, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.645976 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5091: 231378 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:00.646058 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001040646025, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5091, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134007395, "largest_seqno": 134562343, "table_properties": {"data_size": 27028864, "index_size": 6295296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6478584, "raw_average_key_size": 28, "raw_value_size": 27028864, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231378, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001040, "oldest_key_time": 0, "file_creation_time": 1732001040, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5091, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.750987 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5094: 32061 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:00.751066 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001040751033, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5094, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 132746542, "largest_seqno": 132818614, "table_properties": {"data_size": 32602304, "index_size": 921536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641220, "raw_average_key_size": 20, "raw_value_size": 32602293, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32061, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001040, "oldest_key_time": 0, "file_creation_time": 1732001040, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5094, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.953974 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5096: 32061 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:00.954049 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001040954016, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5096, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 132727823, "largest_seqno": 132827835, "table_properties": {"data_size": 32604032, "index_size": 920064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641220, "raw_average_key_size": 20, "raw_value_size": 32604008, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32061, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001040, "oldest_key_time": 0, "file_creation_time": 1732001040, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5096, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:00.955395 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-02 2024/11/19-15:24:00.955522 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00195/att-02/compact.done) = No such file or directory: 11.024 sec on http probe failed: 4, retry 2024/11/19-15:24:01.053337 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5095: 231315 keys, 33325187 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:01.053398 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001041053375, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5095, "file_size": 33325187, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134191053, "largest_seqno": 134871338, "table_properties": {"data_size": 27030720, "index_size": 6293632, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6476820, "raw_average_key_size": 28, "raw_value_size": 27030702, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231315, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001040, "oldest_key_time": 0, "file_creation_time": 1732001040, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5095, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:01.155944 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5097: 32106 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:01.156030 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001041155994, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5097, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 132770669, "largest_seqno": 132832785, "table_properties": {"data_size": 32651008, "index_size": 873472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642120, "raw_average_key_size": 20, "raw_value_size": 32650982, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32106, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001040, "oldest_key_time": 0, "file_creation_time": 1732001040, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5097, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:01.296048 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-01 2024/11/19-15:24:01.296157 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00201/att-01/compact.done) = No such file or directory: 1.570 sec on http probe failed: 1, retry 2024/11/19-15:24:01.321500 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005048.sst): old prealloc_size = 147639500, real_size = 36269088 2024/11/19-15:24:01.321971 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005048.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924222 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1080 GB zip-val = 0.0343 GB avg-val = 116.82 avg-zval = 37.10 2024/11/19-15:24:01.322039 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005048.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 1.688 sec, Key+Value = 0.134 GB 2024/11/19-15:24:01.359902 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5098: 231490 keys, 33324931 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:01.359968 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001041359944, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5098, "file_size": 33324931, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134871339, "largest_seqno": 135460043, "table_properties": {"data_size": 27026048, "index_size": 6298048, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481720, "raw_average_key_size": 28, "raw_value_size": 27025988, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231490, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001041, "oldest_key_time": 0, "file_creation_time": 1732001041, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5098, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:01.461854 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5099: 32059 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:01.461939 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001041461900, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5099, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 132832786, "largest_seqno": 132914007, "table_properties": {"data_size": 32603072, "index_size": 921280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603052, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001041, "oldest_key_time": 0, "file_creation_time": 1732001041, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5099, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:01.563558 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5100: 231487 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:01.563627 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001041563597, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5100, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 135003025, "largest_seqno": 135560332, "table_properties": {"data_size": 27026432, "index_size": 6297728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481636, "raw_average_key_size": 28, "raw_value_size": 27026414, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231487, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001041, "oldest_key_time": 0, "file_creation_time": 1732001041, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5100, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:01.766676 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5101: 32058 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:01.766743 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001041766719, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5101, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 132914008, "largest_seqno": 133624170, "table_properties": {"data_size": 32602752, "index_size": 920832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602743, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001041, "oldest_key_time": 0, "file_creation_time": 1732001041, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5101, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:01.890898 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005048.sst): second pass time = 1.69's, 63.994'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.57's, 65.203'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3870.577'MB/sec (index lex order gen) rebuild zvType time = 0.47's, 0.495'MB/sec write SST data time = 0.10's, 363.732'MB/sec dict compress time = 0.00's, 1431.833'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 11382.739'MB/sec zip my value time = 1.69's, unzip length = 0.108'GB zip my value throughput = 63.994'MB/sec zip pipeline throughput = 63.994'MB/sec entries = 924222 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.82 avg-zval = 37.10 usrkeys = 924222 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.82 avg-zval = 37.10 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0185 GB value = 0.1080 GB dict = 1.08 MB all = 0.1338 GB } __ZipSize{ index = 0.0017 GB value = 0.0343 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5806 value = 3.1485 dict = 1.00 all = 3.6051 } Zip/UnZip{ index = 0.0945 value = 0.3176 dict = 1.00 all = 0.2774 } ---------------------------- total value len = 11.747938 GB avg = 0.257 KB (by entry num) total key len = 1.176231 GB avg = 0.026 KB total ukey len = 0.810575 GB avg = 0.018 KB total ukey num = 0.045707038 Billion total entry num = 0.045707038 Billion write speed all = 39.776129940 MB/sec (with seq num) write speed all = 38.650766038 MB/sec (without seq num) 2024/11/19-15:24:01.944585 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-01 2024/11/19-15:24:01.944678 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00200/att-01/compact.done) = No such file or directory: 9.025 sec on http probe failed: 3, retry 2024/11/19-15:24:02.072595 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5102: 231289 keys, 33325187 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.072659 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001042072635, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5102, "file_size": 33325187, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 135482190, "largest_seqno": 136072432, "table_properties": {"data_size": 27032000, "index_size": 6292352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6476092, "raw_average_key_size": 28, "raw_value_size": 27031950, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231289, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001041, "oldest_key_time": 0, "file_creation_time": 1732001041, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5102, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:02.173861 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5103: 32147 keys, 33524224 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.173944 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001042173903, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5103, "file_size": 33524224, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 133624171, "largest_seqno": 133737039, "table_properties": {"data_size": 32694720, "index_size": 828672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642940, "raw_average_key_size": 20, "raw_value_size": 32694716, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32147, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001041, "oldest_key_time": 0, "file_creation_time": 1732001041, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5103, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:02.272507 139851179493120 [compaction_job.cc:1948] [default] [JOB 194] Generated table #5048: 924222 keys, 37125423 bytes, temperature: kUnknown 2024/11/19-15:24:02.272574 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001042272547, "cf_name": "default", "job": 194, "event": "table_file_creation", "file_number": 5048, "file_size": 37125423, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34290832, "index_size": 1747008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25878216, "raw_average_key_size": 28, "raw_value_size": 107963608, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 924222, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001039, "oldest_key_time": 0, "file_creation_time": 1732001039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5048, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:02.477538 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5104: 231430 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.477598 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001042477575, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5104, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 136000754, "largest_seqno": 136558693, "table_properties": {"data_size": 27028032, "index_size": 6296192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480040, "raw_average_key_size": 28, "raw_value_size": 27028032, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231430, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5104, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:02.561276 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-01 2024/11/19-15:24:02.561383 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00201/att-01/compact.done) = No such file or directory: 2.835 sec on http probe failed: 2, retry 2024/11/19-15:24:02.580092 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5105: 32057 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.580170 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001042580135, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5105, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 133737040, "largest_seqno": 133799105, "table_properties": {"data_size": 32601408, "index_size": 922944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601386, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5105, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:02.672776 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5106: 231473 keys, 33324931 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.672843 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001042672814, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5106, "file_size": 33324931, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 136095087, "largest_seqno": 136684955, "table_properties": {"data_size": 27024704, "index_size": 6299392, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481244, "raw_average_key_size": 28, "raw_value_size": 27024658, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231473, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5106, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:02.672854 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5107: 32055 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.672890 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001042672868, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5107, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 133725828, "largest_seqno": 133825841, "table_properties": {"data_size": 32598592, "index_size": 925568, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32598538, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5107, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:02.714533 139851179493120 [compaction_job.cc:703] job-00194: subcompact[0], size: 0.074332 G, files: /storage/mytopling/.rocksdb [005012,005048] 2024/11/19-15:24:02.717467 139851179493120 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 8 level-0 files rate 5858603 2024/11/19-15:24:02.717730 139851179493120 (Original Log Time 2024/11/19-15:23:56.388832) EVENT_LOG_v1 {"time_micros": 1732001036388816, "job": 194, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4968, 4969, 4970, 4971], "score": 2.60155, "input_data_size": 221704736, "oldest_snapshot_seqno": -1} 2024/11/19-15:24:02.717743 139851179493120 (Original Log Time 2024/11/19-15:24:02.717034) [compaction_job.cc:2022] [default] [JOB 194] Compacted 4@2 files to L3 => 74332112 bytes 2024/11/19-15:24:02.717748 139851179493120 (Original Log Time 2024/11/19-15:24:02.717419) [compaction_job.cc:1224] [default] compacted to: files[8 35 69 44 0 0 0] max score 2.03 (54 files need compaction), MB/sec: 35.0 rd, 11.8 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(211.4, 0.0 +0.0 blob) out(70.9 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 1851089, records dropped: 0 output_compression: Snappy 2024/11/19-15:24:02.717778 139851179493120 (Original Log Time 2024/11/19-15:24:02.717457) EVENT_LOG_v1 {"time_micros": 1732001042717438, "cf": "default", "job": 194, "event": "compaction_finished", "compaction_time_micros": 6325729, "compaction_time_cpu_micros": 4310522, "output_level": 3, "num_output_files": 2, "total_output_size": 74332112, "num_input_records": 1851089, "num_output_records": 1851089, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [8, 35, 69, 44, 0, 0, 0]} 2024/11/19-15:24:02.720652 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5109: 32152 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.720716 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001042720686, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5109, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 133751153, "largest_seqno": 133813313, "table_properties": {"data_size": 32697728, "index_size": 826560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643040, "raw_average_key_size": 20, "raw_value_size": 32697669, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32152, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5109, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:02.720779 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004958.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:02.720809 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001042720806, "job": 194, "event": "table_file_deletion", "file_number": 4958} 2024/11/19-15:24:02.724308 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004962.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:02.724333 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001042724330, "job": 194, "event": "table_file_deletion", "file_number": 4962} 2024/11/19-15:24:02.724482 139851179493120 [compaction_job.cc:2437] [default] [JOB 204] Compacting 4@2 files to L3, score 2.03, subcompactions 1 : 1 2024/11/19-15:24:02.724494 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 202 Base level 2, inputs: [4623(61MB) 4624(62MB) 4625(62MB) 4783(62MB)] 2024/11/19-15:24:02.756840 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5110: 32054 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.756897 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001042756874, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5110, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 133813314, "largest_seqno": 133898310, "table_properties": {"data_size": 32599680, "index_size": 924224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641080, "raw_average_key_size": 20, "raw_value_size": 32599673, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5110, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:02.788443 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5111: 32057 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.788500 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001042788476, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5111, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 133898311, "largest_seqno": 134614664, "table_properties": {"data_size": 32600576, "index_size": 923456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32600563, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5111, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:02.806493 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5108: 231390 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.806548 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001042806526, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5108, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 136684956, "largest_seqno": 137314091, "table_properties": {"data_size": 27028096, "index_size": 6296128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6478920, "raw_average_key_size": 28, "raw_value_size": 27028040, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231390, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5108, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:02.819453 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5112: 32112 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.819501 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001042819481, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5112, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134614665, "largest_seqno": 134717624, "table_properties": {"data_size": 32658176, "index_size": 866112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642240, "raw_average_key_size": 20, "raw_value_size": 32658141, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32112, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5112, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:02.850212 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5114: 32057 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.850265 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001042850244, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5114, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134717625, "largest_seqno": 134779690, "table_properties": {"data_size": 32601280, "index_size": 922368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601226, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5114, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:02.880959 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5115: 32057 keys, 33524288 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.881010 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001042880989, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5115, "file_size": 33524288, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134723929, "largest_seqno": 134813940, "table_properties": {"data_size": 32600704, "index_size": 922752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32600697, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5115, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:02.912724 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5116: 32152 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.912781 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001042912757, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5116, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134731740, "largest_seqno": 134793900, "table_properties": {"data_size": 32696832, "index_size": 827328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643040, "raw_average_key_size": 20, "raw_value_size": 32696812, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32152, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5116, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:02.916322 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5113: 231320 keys, 33325187 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.916368 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001042916346, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5113, "file_size": 33325187, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137002303, "largest_seqno": 137556909, "table_properties": {"data_size": 27030912, "index_size": 6293440, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6476960, "raw_average_key_size": 28, "raw_value_size": 27030904, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231320, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5113, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:02.943839 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5117: 32052 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.943888 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001042943866, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5117, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134793901, "largest_seqno": 134888977, "table_properties": {"data_size": 32597440, "index_size": 926976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641040, "raw_average_key_size": 20, "raw_value_size": 32597390, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32052, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5117, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:02.960607 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-02 2024/11/19-15:24:02.960712 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00195/att-02/compact.done) = No such file or directory: 13.038 sec on http probe failed: 5, giveup 2024/11/19-15:24:02.960741 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00195/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.640 sec, real 13.320 (submit 277.020 ms wait 13.038 s), input 255.640 MiB, send shutdown 2024/11/19-15:24:02.969624 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00195/att-02 2024/11/19-15:24:02.969872 139851148035840 [dcompact_etcd.cc:1773] job-00195/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:02.970163 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00195) = success 2024/11/19-15:24:02.970175 139851148035840 [dcompact_etcd.cc:1773] job-00195/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:02.970202 139851148035840 (Original Log Time 2024/11/19-15:23:22.969708) EVENT_LOG_v1 {"time_micros": 1732001002969702, "job": 195, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4972, 4973, 4974, 4975], "score": 2.43512, "input_data_size": 221714464, "oldest_snapshot_seqno": -1} 2024/11/19-15:24:02.970213 139851148035840 [compaction_job.cc:2437] [default] [JOB 195] Compacting 4@2 files to L3, score 2.44, subcompactions 1 : 1 2024/11/19-15:24:02.970225 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 193 Base level 2, inputs: [4972(52MB) 4973(52MB) 4974(52MB) 4975(52MB)] 2024/11/19-15:24:02.970533 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005120.sst) /storage/mytopling/..//tmp/Topling-82038-kEPUvP 2024/11/19-15:24:02.978832 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5119: 32054 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:02.978891 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001042978861, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5119, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134888978, "largest_seqno": 135602146, "table_properties": {"data_size": 32598912, "index_size": 924992, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641080, "raw_average_key_size": 20, "raw_value_size": 32598851, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5119, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.012902 139851179493120 [dcompact_etcd.cc:1163] job-00204/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.288 sec, input = 253.794 MiB 2024/11/19-15:24:03.019566 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5121: 32151 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.019617 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043019596, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5121, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 135602147, "largest_seqno": 135677244, "table_properties": {"data_size": 32697344, "index_size": 826496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32697281, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5121, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.033707 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5118: 231480 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.033774 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001043033741, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5118, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137342568, "largest_seqno": 137976091, "table_properties": {"data_size": 27025920, "index_size": 6298240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481440, "raw_average_key_size": 28, "raw_value_size": 27025860, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231480, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5118, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.051285 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5122: 32057 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.051337 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043051315, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5122, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 135677245, "largest_seqno": 135760138, "table_properties": {"data_size": 32600512, "index_size": 923648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32600499, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5122, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.067740 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-02 2024/11/19-15:24:03.067858 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00198/att-02/compact.done) = No such file or directory: 4.613 sec on http probe failed: 1, retry 2024/11/19-15:24:03.082903 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5124: 32053 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.082965 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043082944, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5124, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 135721858, "largest_seqno": 135821870, "table_properties": {"data_size": 32598976, "index_size": 924928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32598936, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5124, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.114512 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5125: 32119 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.114568 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043114546, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5125, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 135722188, "largest_seqno": 135784315, "table_properties": {"data_size": 32664768, "index_size": 859072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642380, "raw_average_key_size": 20, "raw_value_size": 32664734, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32119, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5125, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.146183 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5126: 32151 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.146237 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043146216, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5126, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 135784316, "largest_seqno": 135845478, "table_properties": {"data_size": 32696960, "index_size": 826688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32696939, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5126, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.154781 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5123: 231476 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.154850 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001043154819, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5123, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137976092, "largest_seqno": 138555074, "table_properties": {"data_size": 27026752, "index_size": 6297408, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481328, "raw_average_key_size": 28, "raw_value_size": 27026704, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231476, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5123, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.177368 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5127: 32060 keys, 33525376 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.177422 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043177400, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5127, "file_size": 33525376, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 135845479, "largest_seqno": 135948517, "table_properties": {"data_size": 32603136, "index_size": 921408, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32603079, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5127, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.208890 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5129: 32058 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.208955 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043208933, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5129, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 135948518, "largest_seqno": 136659202, "table_properties": {"data_size": 32604160, "index_size": 919936, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32604150, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5129, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.240911 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5130: 32097 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.240973 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043240950, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5130, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 136659203, "largest_seqno": 136751063, "table_properties": {"data_size": 32642368, "index_size": 881984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641940, "raw_average_key_size": 20, "raw_value_size": 32642322, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32097, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5130, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.272231 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5131: 32058 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.272286 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043272263, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5131, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 136751064, "largest_seqno": 136813132, "table_properties": {"data_size": 32603200, "index_size": 921216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603179, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5131, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.275803 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5128: 231261 keys, 33325187 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.275866 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001043275835, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5128, "file_size": 33325187, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137997673, "largest_seqno": 138577538, "table_properties": {"data_size": 27032512, "index_size": 6291840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6475308, "raw_average_key_size": 28, "raw_value_size": 27032482, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231261, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5128, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.303646 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5132: 32061 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.303707 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043303685, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5132, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 136720244, "largest_seqno": 136820259, "table_properties": {"data_size": 32602816, "index_size": 920896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641220, "raw_average_key_size": 20, "raw_value_size": 32602762, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32061, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5132, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.318689 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005120.sst): first pass time = 0.35's, 385.530'MB/sec 2024/11/19-15:24:03.318690 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005120.sst: sumWaitingMem = 0.012 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0121 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:24:03.318722 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005120.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.012 GB, dictZip workingMem = 0.0065 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:24:03.335947 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5134: 32137 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.335999 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043335978, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5134, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 136765185, "largest_seqno": 136833335, "table_properties": {"data_size": 32684864, "index_size": 839232, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642740, "raw_average_key_size": 20, "raw_value_size": 32684810, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32137, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5134, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.367693 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5135: 32057 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.367751 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043367729, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5135, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 136833336, "largest_seqno": 136908297, "table_properties": {"data_size": 32603584, "index_size": 920832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32603582, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5135, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.393400 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005120.sst): index pass time = 0.07's, 198.614'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 926931 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:24:03.393906 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5133: 231448 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.393966 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001043393944, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5133, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138577539, "largest_seqno": 139254143, "table_properties": {"data_size": 27027264, "index_size": 6296960, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480544, "raw_average_key_size": 28, "raw_value_size": 27027232, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231448, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5133, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.399206 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5136: 32061 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.399249 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043399229, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5136, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 136908298, "largest_seqno": 137619128, "table_properties": {"data_size": 32603840, "index_size": 920576, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641220, "raw_average_key_size": 20, "raw_value_size": 32603781, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32061, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5136, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.430441 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5138: 32149 keys, 33524352 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.430497 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043430475, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5138, "file_size": 33524352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137619129, "largest_seqno": 137688296, "table_properties": {"data_size": 32696896, "index_size": 826624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642980, "raw_average_key_size": 20, "raw_value_size": 32696838, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32149, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5138, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.461575 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5139: 32059 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.461628 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043461608, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5139, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137688297, "largest_seqno": 137764241, "table_properties": {"data_size": 32602624, "index_size": 921600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32602566, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5139, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.492982 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5140: 32058 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.493037 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043493015, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5140, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137748844, "largest_seqno": 137788847, "table_properties": {"data_size": 32602944, "index_size": 921280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602934, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5140, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.499133 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5137: 231454 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.499186 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001043499165, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5137, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138961515, "largest_seqno": 139587018, "table_properties": {"data_size": 27023872, "index_size": 6300352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480712, "raw_average_key_size": 28, "raw_value_size": 27023852, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231454, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5137, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.524170 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5141: 32096 keys, 33524800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.524220 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043524199, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5141, "file_size": 33524800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137776302, "largest_seqno": 137827408, "table_properties": {"data_size": 32642880, "index_size": 881088, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641920, "raw_average_key_size": 20, "raw_value_size": 32642840, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32096, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5141, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.555277 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5143: 32058 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.555328 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043555307, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5143, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137827409, "largest_seqno": 137929366, "table_properties": {"data_size": 32600896, "index_size": 922944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32600847, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5143, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.586305 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5144: 32055 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.586362 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043586336, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5144, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137929367, "largest_seqno": 138609743, "table_properties": {"data_size": 32601024, "index_size": 922752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32600986, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5144, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.591385 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5142: 204542 keys, 29462787 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.591430 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001043591410, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5142, "file_size": 29462787, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139205897, "largest_seqno": 139713906, "table_properties": {"data_size": 23882560, "index_size": 5579392, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 5727176, "raw_average_key_size": 28, "raw_value_size": 23882548, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 204542, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5142, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.618027 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5145: 32133 keys, 33524352 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.618078 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043618057, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5145, "file_size": 33524352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138609744, "largest_seqno": 138711937, "table_properties": {"data_size": 32680320, "index_size": 843200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642660, "raw_average_key_size": 20, "raw_value_size": 32680259, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32133, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5145, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.622563 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5146: 32053 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.622611 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001043622589, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5146, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 124763645, "largest_seqno": 124823653, "table_properties": {"data_size": 32598400, "index_size": 926080, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32598359, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5146, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.650094 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5147: 32059 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.650146 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043650125, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5147, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138711938, "largest_seqno": 138774910, "table_properties": {"data_size": 32602432, "index_size": 921920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32602373, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5147, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.653364 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5148: 32151 keys, 33524288 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.653407 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001043653387, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5148, "file_size": 33524288, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 124789207, "largest_seqno": 124850371, "table_properties": {"data_size": 32696832, "index_size": 826624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32696832, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5148, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.656752 139849738749696 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5150: 3116 keys, 3308799 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.656792 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001043656773, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5150, "file_size": 3308799, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 124850372, "largest_seqno": 124884485, "table_properties": {"data_size": 3169792, "index_size": 138176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 62320, "raw_average_key_size": 20, "raw_value_size": 3169751, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 3116, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5150, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.683640 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5149: 32052 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.683689 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043683669, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5149, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138717046, "largest_seqno": 138817063, "table_properties": {"data_size": 32597696, "index_size": 926656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641040, "raw_average_key_size": 20, "raw_value_size": 32597642, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32052, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5149, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.715402 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5151: 32150 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.715452 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043715431, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5151, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138726954, "largest_seqno": 138799120, "table_properties": {"data_size": 32697152, "index_size": 826944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32697146, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5151, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.769849 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5152: 32053 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.769905 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043769884, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5152, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138799121, "largest_seqno": 138850185, "table_properties": {"data_size": 32598784, "index_size": 925504, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32598751, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5152, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.816581 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5153: 32055 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.816660 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043816626, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5153, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138850186, "largest_seqno": 139600091, "table_properties": {"data_size": 32600640, "index_size": 923648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32600637, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5153, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.823761 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-01 2024/11/19-15:24:03.823866 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00201/att-01/compact.done) = No such file or directory: 4.100 sec on http probe failed: 3, retry 2024/11/19-15:24:03.867055 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5154: 32074 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.867110 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043867086, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5154, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139600092, "largest_seqno": 139668992, "table_properties": {"data_size": 32619840, "index_size": 904064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641480, "raw_average_key_size": 20, "raw_value_size": 32619779, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32074, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5154, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.899046 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5155: 32054 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.899103 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043899081, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5155, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139668993, "largest_seqno": 139745108, "table_properties": {"data_size": 32598912, "index_size": 924800, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641080, "raw_average_key_size": 20, "raw_value_size": 32598900, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5155, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.930717 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5156: 32057 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.930772 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043930750, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5156, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139724812, "largest_seqno": 139794820, "table_properties": {"data_size": 32600768, "index_size": 922816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32600722, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5156, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:03.952425 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-01 2024/11/19-15:24:03.952529 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00200/att-01/compact.done) = No such file or directory: 11.031 sec on http probe failed: 4, retry 2024/11/19-15:24:03.977051 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5157: 32099 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:03.977107 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001043977084, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5157, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139727163, "largest_seqno": 139799272, "table_properties": {"data_size": 32643328, "index_size": 881152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641980, "raw_average_key_size": 20, "raw_value_size": 32643274, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32099, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5157, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:04.009502 139849728263936 [compaction_job.cc:1948] [default] [JOB 202] Generated table #5158: 15550 keys, 16230207 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:04.009579 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001044009547, "cf_name": "default", "job": 202, "event": "table_file_creation", "file_number": 5158, "file_size": 16230207, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139799273, "largest_seqno": 139814825, "table_properties": {"data_size": 15814400, "index_size": 414976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 311000, "raw_average_key_size": 20, "raw_value_size": 15814369, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 15550, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001043, "oldest_key_time": 0, "file_creation_time": 1732001043, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5158, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:24:04.011650 139851137550080 [compaction_job.cc:703] job-00202: subcompact[0], size: 0.409630 G, files: /storage/mytopling/.rocksdb [005015,005018,005019,005021,005023,005026,005030,005032,005034,005037,005040,005042,005044,005047,005049] 2024/11/19-15:24:04.011683 139851137550080 [compaction_job.cc:703] job-00202: subcompact[1], size: 0.952552 G, files: /storage/mytopling/.rocksdb [005016,005024,005028,005038,005046,005054,005063,005067,005072,005076,005080,005084,005088,005091,005095,005098,005100,005102,005104,005106,005108,005113,005118,005123,005128,005133,005137,005142,005146,005148,005150] 2024/11/19-15:24:04.011724 139851137550080 [compaction_job.cc:703] job-00202: subcompact[2], size: 3.134041 G, files: /storage/mytopling/.rocksdb [005017,005020,005022,005025,005027,005029,005031,005033,005035,005036,005039,005041,005043,005045,005050,005051,005052,005053,005055,005056,005057,005060,005061,005062,005064,005065,005066,005068,005069,005070,005071,005073,005074,005075,005077,005078,005079,005081,005082,005083,005085,005086,005087,005089,005090,005092,005093,005094,005096,005097,005099,005101,005103,005105,005107,005109,005110,005111,005112,005114,005115,005116,005117,005119,005121,005122,005124,005125,005126,005127,005129,005130,005131,005132,005134,005135,005136,005138,005139,005140,005141,005143,005144,005145,005147,005149,005151,005152,005153,005154,005155,005156,005157,005158] 2024/11/19-15:24:04.393787 139851137550080 (Original Log Time 2024/11/19-15:23:58.250425) EVENT_LOG_v1 {"time_micros": 1732001038250413, "job": 202, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [5004, 4985, 4977, 4966, 4961, 4943, 4904], "files_L1": [4891, 4892, 4894, 4895, 4897, 4898, 4900, 4901, 4902, 4906, 4907, 4909, 4910, 4911, 4913, 4914, 4915, 4917, 4918, 4919, 4921, 4922, 4923, 4924, 4925, 4926, 4927, 4928, 4929, 4930, 4931, 4932, 4933, 4934, 4935], "score": 3.25044, "input_data_size": 4959189110, "oldest_snapshot_seqno": 141230245} 2024/11/19-15:24:04.393808 139851137550080 (Original Log Time 2024/11/19-15:24:04.392115) [compaction_job.cc:2022] [default] [JOB 202] Compacted 7@0 + 35@1 files to L1 => 4496223691 bytes 2024/11/19-15:24:04.393814 139851137550080 (Original Log Time 2024/11/19-15:24:04.393219) [compaction_job.cc:1224] [default] compacted to: files[1 140 69 44 0 0 0] max score 8.38 (190 files need compaction), MB/sec: 860.8 rd, 780.4 wr, level 1, files in(7, 35) out(140 +0 blob) MB in(3616.8, 1112.7 +0.0 blob) out(4287.9 +0.0 blob), read-write-amplify(2.5) write-amplify(1.2) OK, records in: 10727550, records dropped: 0 output_compression: Snappy 2024/11/19-15:24:04.393822 139851137550080 (Original Log Time 2024/11/19-15:24:04.393320) EVENT_LOG_v1 {"time_micros": 1732001044393240, "cf": "default", "job": 202, "event": "compaction_finished", "compaction_time_micros": 5761304, "compaction_time_cpu_micros": 7964019, "output_level": 1, "num_output_files": 140, "total_output_size": 4496223691, "num_input_records": 10727550, "num_output_records": 10727550, "num_subcompactions": 3, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 140, 69, 44, 0, 0, 0]} 2024/11/19-15:24:04.393951 139851137550080 [compaction_job.cc:2437] [default] [JOB 205] Compacting 4@1 files to L2, score 8.38, subcompactions 1 : 1 2024/11/19-15:24:04.393966 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 203 Base level 1, inputs: [5015(6963B) 5018(32KB) 5019(31MB) 5021(31MB)] 2024/11/19-15:24:04.697188 139851137550080 [dcompact_etcd.cc:1163] job-00205/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.303 sec, input = 63.647 MiB 2024/11/19-15:24:04.816979 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005120.sst): old prealloc_size = 147639500, real_size = 36370608 2024/11/19-15:24:04.817411 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005120.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926931 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.80 avg-zval = 37.10 2024/11/19-15:24:04.817476 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005120.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 1.499 sec, Key+Value = 0.134 GB 2024/11/19-15:24:05.076573 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-02 2024/11/19-15:24:05.076659 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00198/att-02/compact.done) = No such file or directory: 6.619 sec on http probe failed: 2, retry 2024/11/19-15:24:05.085179 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-01 2024/11/19-15:24:05.085254 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00201/att-01/compact.done) = No such file or directory: 5.363 sec on http probe failed: 4, retry 2024/11/19-15:24:05.168664 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-02 2024/11/19-15:24:05.168767 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00199/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:24:05.384487 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005120.sst): second pass time = 1.50's, 72.262'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.57's, 65.602'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 4234.430'MB/sec (index lex order gen) rebuild zvType time = 0.47's, 0.492'MB/sec write SST data time = 0.10's, 385.989'MB/sec dict compress time = 0.00's, 1601.798'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 15420.455'MB/sec zip my value time = 1.50's, unzip length = 0.108'GB zip my value throughput = 72.262'MB/sec zip pipeline throughput = 72.262'MB/sec entries = 926931 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.80 avg-zval = 37.10 usrkeys = 926931 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.80 avg-zval = 37.10 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5818 value = 3.1484 dict = 1.00 all = 3.6052 } Zip/UnZip{ index = 0.0945 value = 0.3176 dict = 1.00 all = 0.2774 } ---------------------------- total value len = 11.856201 GB avg = 0.254 KB (by entry num) total key len = 1.202185 GB avg = 0.026 KB total ukey len = 0.829113 GB avg = 0.018 KB total ukey num = 0.046633969 Billion total entry num = 0.046633969 Billion write speed all = 39.761685590 MB/sec (with seq num) write speed all = 38.625713545 MB/sec (without seq num) 2024/11/19-15:24:05.385707 139851148035840 [compaction_job.cc:1948] [default] [JOB 195] Generated table #5120: 926931 keys, 37228697 bytes, temperature: kUnknown 2024/11/19-15:24:05.385759 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001045385735, "cf_name": "default", "job": 195, "event": "table_file_creation", "file_number": 5120, "file_size": 37228697, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34386784, "index_size": 1751936, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954068, "raw_average_key_size": 28, "raw_value_size": 108263682, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926931, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001042, "oldest_key_time": 0, "file_creation_time": 1732001042, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5120, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:05.406780 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005159.sst) /storage/mytopling/..//tmp/Topling-82038-gC1eKm 2024/11/19-15:24:05.726021 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005159.sst): first pass time = 0.32's, 419.269'MB/sec 2024/11/19-15:24:05.726024 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005159.sst: sumWaitingMem = 0.012 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0120 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:24:05.726058 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005159.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.012 GB, dictZip workingMem = 0.0065 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:24:05.795643 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005159.sst): index pass time = 0.07's, 212.560'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 924350 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:24:05.960833 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-01 2024/11/19-15:24:05.960964 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00200/att-01/compact.done) = No such file or directory: 13.038 sec on http probe failed: 5, giveup 2024/11/19-15:24:05.961001 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00200/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.783 sec, real 13.338 (submit 290.804 ms wait 13.038 s), input 253.783 MiB, send shutdown 2024/11/19-15:24:05.969961 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-01 2024/11/19-15:24:05.970211 139851169007360 [dcompact_etcd.cc:1773] job-00200/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:06.273509 139851169007360 [dcompact_etcd.cc:1163] job-00200/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.303 sec, input = 253.783 MiB 2024/11/19-15:24:06.299235 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-00 2024/11/19-15:24:06.299322 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00205/att-00/compact.done) = No such file or directory: 1.591 sec on http probe failed: 1, retry 2024/11/19-15:24:06.346630 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-01 2024/11/19-15:24:06.346730 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00201/att-01/compact.done) = No such file or directory: 6.624 sec on http probe failed: 5, giveup 2024/11/19-15:24:06.346754 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00201/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 62.811 sec, real 6.882 (submit 252.852 ms wait 6.624 s), input 62.811 MiB, send shutdown 2024/11/19-15:24:06.353359 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-01 2024/11/19-15:24:06.353620 139851127064320 [dcompact_etcd.cc:1773] job-00201/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:06.661755 139851127064320 [dcompact_etcd.cc:1163] job-00201/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.308 sec, input = 62.811 MiB 2024/11/19-15:24:07.082256 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-02 2024/11/19-15:24:07.082341 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00198/att-02/compact.done) = No such file or directory: 8.628 sec on http probe failed: 3, retry 2024/11/19-15:24:07.176983 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-02 2024/11/19-15:24:07.177067 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00199/att-02/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:24:07.237347 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005159.sst): old prealloc_size = 147639500, real_size = 36264736 2024/11/19-15:24:07.237821 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005159.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924350 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1080 GB zip-val = 0.0343 GB avg-val = 116.79 avg-zval = 37.08 2024/11/19-15:24:07.237888 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005159.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 1.512 sec, Key+Value = 0.134 GB 2024/11/19-15:24:07.577862 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-00 2024/11/19-15:24:07.577990 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00205/att-00/compact.done) = No such file or directory: 2.875 sec on http probe failed: 2, retry 2024/11/19-15:24:07.771998 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005159.sst): second pass time = 1.51's, 71.436'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.53's, 69.433'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3856.847'MB/sec (index lex order gen) rebuild zvType time = 0.44's, 0.529'MB/sec write SST data time = 0.10's, 381.706'MB/sec dict compress time = 0.00's, 2002.935'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 21289.556'MB/sec zip my value time = 1.51's, unzip length = 0.108'GB zip my value throughput = 71.436'MB/sec zip pipeline throughput = 71.436'MB/sec entries = 924350 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 37.08 usrkeys = 924350 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 37.08 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 10 UnZipSize{ index = 0.0185 GB value = 0.1080 GB dict = 1.08 MB all = 0.1338 GB } __ZipSize{ index = 0.0018 GB value = 0.0343 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5112 value = 3.1498 dict = 1.00 all = 3.6055 } Zip/UnZip{ index = 0.0951 value = 0.3175 dict = 1.00 all = 0.2774 } ---------------------------- total value len = 11.964160 GB avg = 0.252 KB (by entry num) total key len = 1.228067 GB avg = 0.026 KB total ukey len = 0.847600 GB avg = 0.018 KB total ukey num = 0.047558319 Billion total entry num = 0.047558319 Billion write speed all = 39.879304632 MB/sec (with seq num) write speed all = 38.729177199 MB/sec (without seq num) 2024/11/19-15:24:07.775968 139851148035840 [compaction_job.cc:1948] [default] [JOB 195] Generated table #5159: 924350 keys, 37121453 bytes, temperature: kUnknown 2024/11/19-15:24:07.776022 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001047775998, "cf_name": "default", "job": 195, "event": "table_file_creation", "file_number": 5159, "file_size": 37121453, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34274704, "index_size": 1758784, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25881800, "raw_average_key_size": 28, "raw_value_size": 107958420, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 924350, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001045, "oldest_key_time": 0, "file_creation_time": 1732001045, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5159, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:07.797299 139851148035840 [compaction_job.cc:703] job-00195: subcompact[0], size: 0.074350 G, files: /storage/mytopling/.rocksdb [005120,005159] 2024/11/19-15:24:07.799784 139851148035840 (Original Log Time 2024/11/19-15:24:02.970265) EVENT_LOG_v1 {"time_micros": 1732001042970242, "job": 195, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4972, 4973, 4974, 4975], "score": 2.43512, "input_data_size": 221714464, "oldest_snapshot_seqno": -1} 2024/11/19-15:24:07.799795 139851148035840 (Original Log Time 2024/11/19-15:24:07.799293) [compaction_job.cc:2022] [default] [JOB 195] Compacted 4@2 files to L3 => 74350150 bytes 2024/11/19-15:24:07.799798 139851148035840 (Original Log Time 2024/11/19-15:24:07.799654) [compaction_job.cc:1224] [default] compacted to: files[1 140 65 46 0 0 0] max score 8.26 (186 files need compaction), MB/sec: 45.9 rd, 15.4 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(211.4, 0.0 +0.0 blob) out(70.9 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 1851281, records dropped: 0 output_compression: Snappy 2024/11/19-15:24:07.799814 139851148035840 (Original Log Time 2024/11/19-15:24:07.799681) EVENT_LOG_v1 {"time_micros": 1732001047799667, "cf": "default", "job": 195, "event": "compaction_finished", "compaction_time_micros": 4827049, "compaction_time_cpu_micros": 3876296, "output_level": 3, "num_output_files": 2, "total_output_size": 74350150, "num_input_records": 1851281, "num_output_records": 1851281, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 140, 65, 46, 0, 0, 0]} 2024/11/19-15:24:07.803965 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004873.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:07.803991 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001047803988, "job": 195, "event": "table_file_deletion", "file_number": 4873} 2024/11/19-15:24:07.808337 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004875.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:07.808360 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001047808357, "job": 195, "event": "table_file_deletion", "file_number": 4875} 2024/11/19-15:24:07.813490 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004877.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:07.813512 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001047813510, "job": 195, "event": "table_file_deletion", "file_number": 4877} 2024/11/19-15:24:07.818258 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004878.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:07.818279 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001047818277, "job": 195, "event": "table_file_deletion", "file_number": 4878} 2024/11/19-15:24:07.820184 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004964.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:07.820202 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001047820200, "job": 195, "event": "table_file_deletion", "file_number": 4964} 2024/11/19-15:24:07.820360 139851148035840 [compaction_job.cc:2437] [default] [JOB 206] Compacting 4@1 files to L2, score 8.26, subcompactions 1 : 1 2024/11/19-15:24:07.820370 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 204 Base level 1, inputs: [5023(31MB) 5026(31MB) 5030(31MB) 5032(31MB)] 2024/11/19-15:24:08.022380 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-00 2024/11/19-15:24:08.022494 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00204/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:24:08.113100 139851148035840 [dcompact_etcd.cc:1163] job-00206/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.292 sec, input = 127.235 MiB 2024/11/19-15:24:08.237530 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-02 2024/11/19-15:24:08.237638 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00201/att-02/compact.done) = No such file or directory: 1.570 sec on http probe failed: 1, retry 2024/11/19-15:24:08.859846 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-00 2024/11/19-15:24:08.859956 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00205/att-00/compact.done) = No such file or directory: 4.154 sec on http probe failed: 3, retry 2024/11/19-15:24:09.088205 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-02 2024/11/19-15:24:09.088302 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00198/att-02/compact.done) = No such file or directory: 10.634 sec on http probe failed: 4, retry 2024/11/19-15:24:09.182343 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-02 2024/11/19-15:24:09.182449 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00199/att-02/compact.done) = No such file or directory: 9.018 sec on http probe failed: 3, retry 2024/11/19-15:24:09.502379 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-02 2024/11/19-15:24:09.502456 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00201/att-02/compact.done) = No such file or directory: 2.832 sec on http probe failed: 2, retry 2024/11/19-15:24:10.031345 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-00 2024/11/19-15:24:10.031473 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00204/att-00/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:24:10.141434 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-00 2024/11/19-15:24:10.141514 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00205/att-00/compact.done) = No such file or directory: 5.436 sec on http probe failed: 4, retry 2024/11/19-15:24:10.767493 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-02 2024/11/19-15:24:10.767593 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00201/att-02/compact.done) = No such file or directory: 4.097 sec on http probe failed: 3, retry 2024/11/19-15:24:11.096826 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-02 2024/11/19-15:24:11.096933 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00198/att-02/compact.done) = No such file or directory: 12.640 sec on http probe failed: 5, giveup 2024/11/19-15:24:11.096960 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00198/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 184.525 sec, real 12.987 (submit 338.155 ms wait 12.640 s), input 184.525 MiB, send shutdown 2024/11/19-15:24:11.107301 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00198/att-02 2024/11/19-15:24:11.107563 139850978166528 [dcompact_etcd.cc:1773] job-00198/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:11.107856 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00198) = success 2024/11/19-15:24:11.107867 139850978166528 [dcompact_etcd.cc:1773] job-00198/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:11.107891 139850978166528 (Original Log Time 2024/11/19-15:23:32.219102) EVENT_LOG_v1 {"time_micros": 1732001012219095, "job": 198, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4978, 4979, 4967], "score": 2.3513, "input_data_size": 160039432, "oldest_snapshot_seqno": 139877616} 2024/11/19-15:24:11.107901 139850978166528 [compaction_job.cc:2437] [default] [JOB 198] Compacting 3@2 files to L3, score 2.35, subcompactions 1 : 1 2024/11/19-15:24:11.107910 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 196 Base level 2, inputs: [4978(52MB) 4979(52MB) 4967(46MB)] 2024/11/19-15:24:11.108233 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005160.sst) /storage/mytopling/..//tmp/Topling-82038-cWtIDk 2024/11/19-15:24:11.192995 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-02 2024/11/19-15:24:11.193103 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00199/att-02/compact.done) = No such file or directory: 11.023 sec on http probe failed: 4, retry 2024/11/19-15:24:11.278807 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-02 2024/11/19-15:24:11.278894 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00200/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:24:11.303731 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-00 2024/11/19-15:24:11.303839 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00206/att-00/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:24:11.423341 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-00 2024/11/19-15:24:11.423448 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00205/att-00/compact.done) = No such file or directory: 6.717 sec on http probe failed: 5, giveup 2024/11/19-15:24:11.423466 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00205/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 63.647 sec, real 7.029 (submit 302.608 ms wait 6.717 s), input 63.647 MiB, send shutdown 2024/11/19-15:24:11.431607 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-00 2024/11/19-15:24:11.447168 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005160.sst): first pass time = 0.34's, 396.019'MB/sec 2024/11/19-15:24:11.447174 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005160.sst: sumWaitingMem = 0.012 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0121 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:24:11.447203 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005160.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.012 GB, dictZip workingMem = 0.0065 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:24:11.516975 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005160.sst): index pass time = 0.07's, 212.576'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 926947 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:24:11.542023 139851137550080 [dcompact_etcd.cc:1773] job-00205/att-00: CleanFiles 0.110 sec : 0.000 2024/11/19-15:24:11.823996 139851137550080 [dcompact_etcd.cc:1163] job-00205/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.282 sec, input = 63.647 MiB 2024/11/19-15:24:12.029020 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-02 2024/11/19-15:24:12.029113 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00201/att-02/compact.done) = No such file or directory: 5.362 sec on http probe failed: 4, retry 2024/11/19-15:24:12.040583 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-00 2024/11/19-15:24:12.040652 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00204/att-00/compact.done) = No such file or directory: 9.019 sec on http probe failed: 3, retry 2024/11/19-15:24:12.979751 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005160.sst): old prealloc_size = 147639500, real_size = 36371888 2024/11/19-15:24:12.980202 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005160.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926947 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.80 avg-zval = 37.09 2024/11/19-15:24:12.980264 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005160.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 1.533 sec, Key+Value = 0.134 GB 2024/11/19-15:24:13.202552 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-02 2024/11/19-15:24:13.202631 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00199/att-02/compact.done) = No such file or directory: 13.034 sec on http probe failed: 5, giveup 2024/11/19-15:24:13.202649 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00199/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.865 sec, real 13.359 (submit 315.965 ms wait 13.034 s), input 253.865 MiB, send shutdown 2024/11/19-15:24:13.211774 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00199/att-02 2024/11/19-15:24:13.212005 139851158521600 [dcompact_etcd.cc:1773] job-00199/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:13.212244 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00199) = success 2024/11/19-15:24:13.212251 139851158521600 [dcompact_etcd.cc:1773] job-00199/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:13.212272 139851158521600 (Original Log Time 2024/11/19-15:23:33.151610) EVENT_LOG_v1 {"time_micros": 1732001013151604, "job": 199, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4607, 4608, 4609, 4610], "score": 2.23117, "input_data_size": 262232790, "oldest_snapshot_seqno": 139893621} 2024/11/19-15:24:13.212279 139851158521600 [compaction_job.cc:2437] [default] [JOB 199] Compacting 4@2 files to L3, score 2.23, subcompactions 1 : 1 2024/11/19-15:24:13.212286 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 197 Base level 2, inputs: [4607(62MB) 4608(63MB) 4609(61MB) 4610(63MB)] 2024/11/19-15:24:13.215203 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005161.sst) /storage/mytopling/..//tmp/Topling-82038-nBE7y5 2024/11/19-15:24:13.287994 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-02 2024/11/19-15:24:13.288072 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00200/att-02/compact.done) = No such file or directory: 7.005 sec on http probe failed: 2, retry 2024/11/19-15:24:13.294567 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-02 2024/11/19-15:24:13.294639 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00201/att-02/compact.done) = No such file or directory: 6.624 sec on http probe failed: 5, giveup 2024/11/19-15:24:13.294661 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00201/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 62.811 sec, real 6.941 (submit 308.028 ms wait 6.624 s), input 62.811 MiB, send shutdown 2024/11/19-15:24:13.299134 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00201/att-02 2024/11/19-15:24:13.299366 139851127064320 [dcompact_etcd.cc:1773] job-00201/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:13.300665 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00201) = success 2024/11/19-15:24:13.300675 139851127064320 [dcompact_etcd.cc:1773] job-00201/att-03: CleanFiles 0.001 sec : 0.000 2024/11/19-15:24:13.300696 139851127064320 (Original Log Time 2024/11/19-15:23:52.513812) EVENT_LOG_v1 {"time_micros": 1732001032513804, "job": 201, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4621], "score": 1.98332, "input_data_size": 64799946, "oldest_snapshot_seqno": 140912171} 2024/11/19-15:24:13.300706 139851127064320 [compaction_job.cc:2437] [default] [JOB 201] Compacting 1@2 files to L3, score 1.98, subcompactions 1 : 1 2024/11/19-15:24:13.300715 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 199 Base level 2, inputs: [4621(61MB)] 2024/11/19-15:24:13.300987 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005162.sst) /storage/mytopling/..//tmp/Topling-82038-cmxA2X 2024/11/19-15:24:13.309313 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-00 2024/11/19-15:24:13.309378 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00206/att-00/compact.done) = No such file or directory: 5.191 sec on http probe failed: 2, retry 2024/11/19-15:24:13.425062 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-01 2024/11/19-15:24:13.425193 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00205/att-01/compact.done) = No such file or directory: 1.591 sec on http probe failed: 1, retry 2024/11/19-15:24:13.453533 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005162.sst): first pass time = 0.15's, 431.792'MB/sec 2024/11/19-15:24:13.453531 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005162.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:24:13.453576 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005162.sst: sumWaitingMem = 0.004 GB, sumWorkingMem = 0.000 GB, dictZip workingMem = 0.0040 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:24:13.453646 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005162.sst): index pass time = 0.00's,6951.202'MB/sec index type = UintIndex_AllOne usrkeys = 63514 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:24:13.459574 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005161.sst): first pass time = 0.24's, 549.269'MB/sec 2024/11/19-15:24:13.459580 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005161.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.004 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:24:13.459606 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005161.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.005 GB, dictZip workingMem = 0.0083 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:24:13.466121 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005161.sst): index pass time = 0.01's, 158.921'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 129430 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:24:13.579470 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005160.sst): second pass time = 1.53's, 70.644'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.60's, 62.083'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 4056.694'MB/sec (index lex order gen) rebuild zvType time = 0.49's, 0.478'MB/sec write SST data time = 0.11's, 326.596'MB/sec dict compress time = 0.00's, 1395.596'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18816.560'MB/sec zip my value time = 1.53's, unzip length = 0.108'GB zip my value throughput = 70.644'MB/sec zip pipeline throughput = 70.644'MB/sec entries = 926947 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 37.09 usrkeys = 926947 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 37.09 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5132 value = 3.1493 dict = 1.00 all = 3.6051 } Zip/UnZip{ index = 0.0951 value = 0.3175 dict = 1.00 all = 0.2774 } ---------------------------- total value len = 12.072423 GB avg = 0.249 KB (by entry num) total key len = 1.254021 GB avg = 0.026 KB total ukey len = 0.866139 GB avg = 0.018 KB total ukey num = 0.048485266 Billion total entry num = 0.048485266 Billion write speed all = 39.590011777 MB/sec (with seq num) write speed all = 38.437697049 MB/sec (without seq num) 2024/11/19-15:24:13.617582 139850978166528 [compaction_job.cc:1948] [default] [JOB 198] Generated table #5160: 926947 keys, 37229903 bytes, temperature: kUnknown 2024/11/19-15:24:13.617637 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001053617612, "cf_name": "default", "job": 198, "event": "table_file_creation", "file_number": 5160, "file_size": 37229903, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34376544, "index_size": 1763392, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954516, "raw_average_key_size": 28, "raw_value_size": 108263318, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926947, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001051, "oldest_key_time": 0, "file_creation_time": 1732001051, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5160, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:13.642477 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005163.sst) /storage/mytopling/..//tmp/Topling-82038-6Rm2R7 2024/11/19-15:24:14.055422 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-00 2024/11/19-15:24:14.055588 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00204/att-00/compact.done) = No such file or directory: 11.030 sec on http probe failed: 4, retry 2024/11/19-15:24:14.065443 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005163.sst: sumWaitingMem = 0.005 GB, sumWorkingMem = 0.012 GB, nltTrie workingMem = 0.0053 GB, waited 0.000 sec, Key+Value = 0.059 GB 2024/11/19-15:24:14.065456 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005163.sst): first pass time = 0.42's, 140.139'MB/sec 2024/11/19-15:24:14.068967 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005163.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.018 GB, dictZip workingMem = 0.0029 GB, waited 0.004 sec, Key+Value = 0.059 GB 2024/11/19-15:24:14.156021 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005163.sst): index pass time = 0.09's, 72.374'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 409415 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0066 GB zip-key = 0.0008 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:24:14.688831 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005162.sst): old prealloc_size = 71279940, real_size = 19771960 2024/11/19-15:24:14.691961 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005162.sst): index type = UintIndex_AllOne , store type = DictZipBlobStore usrkeys = 63514 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0646 GB zip-val = 0.0198 GB avg-val =1016.97 avg-zval = 311.04 2024/11/19-15:24:14.708303 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-01 2024/11/19-15:24:14.708466 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00205/att-01/compact.done) = No such file or directory: 2.877 sec on http probe failed: 2, retry 2024/11/19-15:24:14.762573 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005162.sst): second pass time = 1.24's, 52.293'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 277.110'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 0.062'MB/sec (index lex order gen) rebuild zvType time = 0.03's, 0.489'MB/sec write SST data time = 0.04's, 535.761'MB/sec dict compress time = 0.00's, 1256.345'MB/sec z-dict build time = 0.00's, sample length = 0.661'MB, throughput = 11477.479'MB/sec zip my value time = 1.24's, unzip length = 0.065'GB zip my value throughput = 52.293'MB/sec zip pipeline throughput = 95.039'MB/sec entries = 63514 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.97 avg-zval = 311.04 usrkeys = 63514 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.97 avg-zval = 311.04 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 3 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.66 MB all = 0.0659 GB } __ZipSize{ index = 0.0000 GB value = 0.0198 GB dict = 0.66 MB all = 0.0204 GB } UnZip/Zip{ index =3969.6250 value = 3.2695 dict = 1.00 all = 3.2257 } Zip/UnZip{ index = 0.0003 value = 0.3059 dict = 1.00 all = 0.3100 } ---------------------------- total value len = 12.137015 GB avg = 0.250 KB (by entry num) total key len = 1.255292 GB avg = 0.026 KB total ukey len = 0.866901 GB avg = 0.018 KB total ukey num = 0.048548780 Billion total entry num = 0.048548780 Billion write speed all = 39.646333164 MB/sec (with seq num) write speed all = 38.496549981 MB/sec (without seq num) 2024/11/19-15:24:14.770089 139851127064320 [compaction_job.cc:1948] [default] [JOB 201] Generated table #5162: 63514 keys, 20418034 bytes, temperature: kUnknown 2024/11/19-15:24:14.770161 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001054770126, "cf_name": "default", "job": 201, "event": "table_file_creation", "file_number": 5162, "file_size": 20418034, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 19755712, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1270280, "raw_average_key_size": 20, "raw_value_size": 64591983, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63514, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-DictZipBlobStore", "creation_time": 1732001053, "oldest_key_time": 0, "file_creation_time": 1732001053, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5162, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:14.807263 139851127064320 [compaction_job.cc:703] job-00201: subcompact[0], size: 0.020418 G, files: /storage/mytopling/.rocksdb [005162] 2024/11/19-15:24:14.808559 139851127064320 (Original Log Time 2024/11/19-15:24:13.300749) EVENT_LOG_v1 {"time_micros": 1732001053300731, "job": 201, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4621], "score": 1.98332, "input_data_size": 64799946, "oldest_snapshot_seqno": 140912171} 2024/11/19-15:24:14.808574 139851127064320 (Original Log Time 2024/11/19-15:24:14.807884) [compaction_job.cc:2022] [default] [JOB 201] Compacted 1@2 files to L3 => 20418034 bytes 2024/11/19-15:24:14.808579 139851127064320 (Original Log Time 2024/11/19-15:24:14.808414) [compaction_job.cc:1224] [default] compacted to: files[1 140 64 47 0 0 0] max score 8.01 (182 files need compaction), MB/sec: 43.0 rd, 13.6 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(61.8, 0.0 +0.0 blob) out(19.5 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 63514, records dropped: 0 output_compression: Snappy 2024/11/19-15:24:14.808600 139851127064320 (Original Log Time 2024/11/19-15:24:14.808457) EVENT_LOG_v1 {"time_micros": 1732001054808432, "cf": "default", "job": 201, "event": "compaction_finished", "compaction_time_micros": 1506535, "compaction_time_cpu_micros": 401404, "output_level": 3, "num_output_files": 1, "total_output_size": 20418034, "num_input_records": 63514, "num_output_records": 63514, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 140, 64, 47, 0, 0, 0]} 2024/11/19-15:24:14.808726 139851127064320 [compaction_job.cc:2437] [default] [JOB 207] Compacting 4@1 files to L2, score 8.01, subcompactions 1 : 1 2024/11/19-15:24:14.808740 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 205 Base level 1, inputs: [5034(31MB) 5037(31MB) 5040(31MB) 5042(31MB)] 2024/11/19-15:24:15.142340 139851127064320 [dcompact_etcd.cc:1163] job-00207/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.333 sec, input = 127.238 MiB 2024/11/19-15:24:15.299802 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-02 2024/11/19-15:24:15.299948 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00200/att-02/compact.done) = No such file or directory: 9.015 sec on http probe failed: 3, retry 2024/11/19-15:24:15.323912 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-00 2024/11/19-15:24:15.324004 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00206/att-00/compact.done) = No such file or directory: 7.198 sec on http probe failed: 3, retry 2024/11/19-15:24:15.408416 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005163.sst): old prealloc_size = 147639500, real_size = 16223760 2024/11/19-15:24:15.408665 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005163.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 409415 prefix = 4 raw-key = 0.0066 GB zip-key = 0.0008 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0478 GB zip-val = 0.0153 GB avg-val = 116.77 avg-zval = 37.48 2024/11/19-15:24:15.408734 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005163.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.008 GB, reorder workingMem = 0.0010 GB, waited 1.343 sec, Key+Value = 0.059 GB 2024/11/19-15:24:15.433315 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005161.sst): old prealloc_size = 147639500, real_size = 40351616 2024/11/19-15:24:15.433420 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005161.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129430 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1016.99 avg-zval = 309.82 2024/11/19-15:24:15.433485 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005161.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 1.974 sec, Key+Value = 0.134 GB 2024/11/19-15:24:15.555605 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005161.sst): second pass time = 1.97's, 66.692'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 341.045'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2074.144'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 0.368'MB/sec write SST data time = 0.03's, 1217.575'MB/sec dict compress time = 0.00's, 1551.244'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 17850.376'MB/sec zip my value time = 1.97's, unzip length = 0.132'GB zip my value throughput = 66.692'MB/sec zip pipeline throughput = 122.846'MB/sec entries = 129430 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.82 usrkeys = 129430 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.82 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0980 value = 3.2825 dict = 1.00 all = 3.2187 } Zip/UnZip{ index = 0.1409 value = 0.3046 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 12.268644 GB avg = 0.252 KB (by entry num) total key len = 1.257880 GB avg = 0.026 KB total ukey len = 0.868454 GB avg = 0.018 KB total ukey num = 0.048678210 Billion total entry num = 0.048678210 Billion write speed all = 39.949872097 MB/sec (with seq num) write speed all = 38.799724023 MB/sec (without seq num) 2024/11/19-15:24:15.569416 139851158521600 [compaction_job.cc:1948] [default] [JOB 199] Generated table #5161: 129430 keys, 41699351 bytes, temperature: kUnknown 2024/11/19-15:24:15.569488 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001055569455, "cf_name": "default", "job": 199, "event": "table_file_creation", "file_number": 5161, "file_size": 41699351, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 40100272, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588600, "raw_average_key_size": 20, "raw_value_size": 131629360, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129430, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001053, "oldest_key_time": 0, "file_creation_time": 1732001053, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5161, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:15.650912 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005163.sst): second pass time = 1.34's, 35.692'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.24's, 68.511'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3419.819'MB/sec (index lex order gen) rebuild zvType time = 0.20's, 0.520'MB/sec write SST data time = 0.05's, 365.166'MB/sec dict compress time = 0.00's, 689.502'MB/sec z-dict build time = 0.00's, sample length = 0.486'MB, throughput = 4803.820'MB/sec zip my value time = 1.34's, unzip length = 0.048'GB zip my value throughput = 35.692'MB/sec zip pipeline throughput = 79.232'MB/sec entries = 409415 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 37.48 usrkeys = 409415 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 37.48 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0082 GB value = 0.0478 GB dict = 0.49 MB all = 0.0593 GB } __ZipSize{ index = 0.0008 GB value = 0.0153 GB dict = 0.49 MB all = 0.0166 GB } UnZip/Zip{ index = 10.5685 value = 3.1152 dict = 1.00 all = 3.5678 } Zip/UnZip{ index = 0.0946 value = 0.3210 dict = 1.00 all = 0.2803 } ---------------------------- total value len = 12.316451 GB avg = 0.251 KB (by entry num) total key len = 1.269344 GB avg = 0.026 KB total ukey len = 0.876643 GB avg = 0.018 KB total ukey num = 0.049087625 Billion total entry num = 0.049087625 Billion write speed all = 40.113632085 MB/sec (with seq num) write speed all = 38.954136924 MB/sec (without seq num) 2024/11/19-15:24:15.675640 139850978166528 [compaction_job.cc:1948] [default] [JOB 198] Generated table #5163: 409415 keys, 16612663 bytes, temperature: kUnknown 2024/11/19-15:24:15.675749 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001055675701, "cf_name": "default", "job": 198, "event": "table_file_creation", "file_number": 5163, "file_size": 16612663, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 15346432, "index_size": 774784, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 11463620, "raw_average_key_size": 28, "raw_value_size": 47806778, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 409415, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001053, "oldest_key_time": 0, "file_creation_time": 1732001053, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5163, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:15.699481 139850978166528 [compaction_job.cc:703] job-00198: subcompact[0], size: 0.053843 G, files: /storage/mytopling/.rocksdb [005160,005163] 2024/11/19-15:24:15.702124 139850978166528 (Original Log Time 2024/11/19-15:24:11.107957) EVENT_LOG_v1 {"time_micros": 1732001051107936, "job": 198, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4978, 4979, 4967], "score": 2.3513, "input_data_size": 160039432, "oldest_snapshot_seqno": 139877616} 2024/11/19-15:24:15.702135 139850978166528 (Original Log Time 2024/11/19-15:24:15.701629) [compaction_job.cc:2022] [default] [JOB 198] Compacted 3@2 files to L3 => 53842566 bytes 2024/11/19-15:24:15.702139 139850978166528 (Original Log Time 2024/11/19-15:24:15.701997) [compaction_job.cc:1224] [default] compacted to: files[1 140 61 49 0 0 0] max score 7.76 (178 files need compaction), MB/sec: 34.9 rd, 11.7 wr, level 3, files in(3, 0) out(2 +0 blob) MB in(152.6, 0.0 +0.0 blob) out(51.3 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 1336362, records dropped: 0 output_compression: Snappy 2024/11/19-15:24:15.702157 139850978166528 (Original Log Time 2024/11/19-15:24:15.702023) EVENT_LOG_v1 {"time_micros": 1732001055702011, "cf": "default", "job": 198, "event": "compaction_finished", "compaction_time_micros": 4591547, "compaction_time_cpu_micros": 3119670, "output_level": 3, "num_output_files": 2, "total_output_size": 53842566, "num_input_records": 1336362, "num_output_records": 1336362, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 140, 61, 49, 0, 0, 0]} 2024/11/19-15:24:15.730462 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004955.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:15.730495 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001055730491, "job": 198, "event": "table_file_deletion", "file_number": 4955} 2024/11/19-15:24:15.740575 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004957.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:15.740601 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001055740598, "job": 198, "event": "table_file_deletion", "file_number": 4957} 2024/11/19-15:24:15.740755 139850978166528 [compaction_job.cc:2437] [default] [JOB 208] Compacting 4@1 files to L2, score 7.76, subcompactions 1 : 1 2024/11/19-15:24:15.740763 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 206 Base level 1, inputs: [5044(31MB) 5047(17MB) 5049(21MB) 5016(21MB)] 2024/11/19-15:24:15.755123 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005164.sst) /storage/mytopling/..//tmp/Topling-82038-HbzOHY 2024/11/19-15:24:15.922904 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005164.sst): first pass time = 0.17's, 786.692'MB/sec 2024/11/19-15:24:15.922910 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005164.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.132 GB 2024/11/19-15:24:15.922949 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005164.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0081 GB, waited 0.000 sec, Key+Value = 0.132 GB 2024/11/19-15:24:15.929296 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005164.sst): index pass time = 0.01's, 160.462'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 127272 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:24:15.989274 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-01 2024/11/19-15:24:15.989362 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00205/att-01/compact.done) = No such file or directory: 4.157 sec on http probe failed: 3, retry 2024/11/19-15:24:16.058780 139850978166528 [dcompact_etcd.cc:1163] job-00208/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.318 sec, input = 90.065 MiB 2024/11/19-15:24:16.061189 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-00 2024/11/19-15:24:16.061250 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00204/att-00/compact.done) = No such file or directory: 13.043 sec on http probe failed: 5, giveup 2024/11/19-15:24:16.061269 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00204/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.794 sec, real 13.336 (submit 287.790 ms wait 13.043 s), input 253.794 MiB, send shutdown 2024/11/19-15:24:16.070019 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-00 2024/11/19-15:24:16.070283 139851179493120 [dcompact_etcd.cc:1773] job-00204/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:16.303907 139851179493120 [dcompact_etcd.cc:1163] job-00204/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.234 sec, input = 253.794 MiB 2024/11/19-15:24:17.029520 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005164.sst): old prealloc_size = 147639500, real_size = 39680672 2024/11/19-15:24:17.029619 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005164.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 127272 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1294 GB zip-val = 0.0394 GB avg-val =1016.98 avg-zval = 309.84 2024/11/19-15:24:17.029676 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005164.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 1.107 sec, Key+Value = 0.132 GB 2024/11/19-15:24:17.145569 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005164.sst): second pass time = 1.11's, 116.971'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 353.436'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2642.115'MB/sec (index lex order gen) rebuild zvType time = 0.08's, 0.376'MB/sec write SST data time = 0.03's, 1313.837'MB/sec dict compress time = 0.00's, 1983.693'MB/sec z-dict build time = 0.00's, sample length = 1.358'MB, throughput = 19804.321'MB/sec zip my value time = 1.11's, unzip length = 0.129'GB zip my value throughput = 116.971'MB/sec zip pipeline throughput = 116.971'MB/sec entries = 127272 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.98 avg-zval = 309.84 usrkeys = 127272 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1016.98 avg-zval = 309.84 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0015 GB value = 0.1294 GB dict = 1.36 MB all = 0.1320 GB } __ZipSize{ index = 0.0002 GB value = 0.0394 GB dict = 1.36 MB all = 0.0410 GB } UnZip/Zip{ index = 7.0959 value = 3.2823 dict = 1.00 all = 3.2184 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 12.445885 GB avg = 0.253 KB (by entry num) total key len = 1.271889 GB avg = 0.026 KB total ukey len = 0.878170 GB avg = 0.018 KB total ukey num = 0.049214897 Billion total entry num = 0.049214897 Billion write speed all = 40.325353791 MB/sec (with seq num) write speed all = 39.167960080 MB/sec (without seq num) 2024/11/19-15:24:17.146912 139851158521600 [compaction_job.cc:1948] [default] [JOB 199] Generated table #5164: 127272 keys, 41008087 bytes, temperature: kUnknown 2024/11/19-15:24:17.146976 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001057146952, "cf_name": "default", "job": 199, "event": "table_file_creation", "file_number": 5164, "file_size": 41008087, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 39433424, "index_size": 215232, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2545440, "raw_average_key_size": 20, "raw_value_size": 129433410, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 127272, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001055, "oldest_key_time": 0, "file_creation_time": 1732001055, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5164, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:17.169108 139851158521600 [compaction_job.cc:703] job-00199: subcompact[0], size: 0.082707 G, files: /storage/mytopling/.rocksdb [005161,005164] 2024/11/19-15:24:17.170999 139851158521600 (Original Log Time 2024/11/19-15:24:13.212317) EVENT_LOG_v1 {"time_micros": 1732001053212299, "job": 199, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4607, 4608, 4609, 4610], "score": 2.23117, "input_data_size": 262232790, "oldest_snapshot_seqno": 139893621} 2024/11/19-15:24:17.171010 139851158521600 (Original Log Time 2024/11/19-15:24:17.170521) [compaction_job.cc:2022] [default] [JOB 199] Compacted 4@2 files to L3 => 82707438 bytes 2024/11/19-15:24:17.171014 139851158521600 (Original Log Time 2024/11/19-15:24:17.170857) [compaction_job.cc:1224] [default] compacted to: files[1 140 57 51 0 0 0] max score 7.58 (174 files need compaction), MB/sec: 66.3 rd, 20.9 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(250.1, 0.0 +0.0 blob) out(78.9 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 256702, records dropped: 0 output_compression: Snappy 2024/11/19-15:24:17.171029 139851158521600 (Original Log Time 2024/11/19-15:24:17.170885) EVENT_LOG_v1 {"time_micros": 1732001057170871, "cf": "default", "job": 199, "event": "compaction_finished", "compaction_time_micros": 3956806, "compaction_time_cpu_micros": 1138292, "output_level": 3, "num_output_files": 2, "total_output_size": 82707438, "num_input_records": 256702, "num_output_records": 256702, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 140, 57, 51, 0, 0, 0]} 2024/11/19-15:24:17.171958 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004879.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:17.171980 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001057171978, "job": 199, "event": "table_file_deletion", "file_number": 4879} 2024/11/19-15:24:17.172878 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004881.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:17.172892 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001057172890, "job": 199, "event": "table_file_deletion", "file_number": 4881} 2024/11/19-15:24:17.173779 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004883.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:17.173793 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001057173792, "job": 199, "event": "table_file_deletion", "file_number": 4883} 2024/11/19-15:24:17.174728 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004884.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:17.174743 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001057174742, "job": 199, "event": "table_file_deletion", "file_number": 4884} 2024/11/19-15:24:17.174888 139851158521600 [compaction_job.cc:2437] [default] [JOB 209] Compacting 4@1 files to L2, score 7.58, subcompactions 1 : 1 2024/11/19-15:24:17.174897 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 207 Base level 1, inputs: [5024(4437KB) 5028(24MB) 5038(31MB) 5046(31MB)] 2024/11/19-15:24:17.272427 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-01 2024/11/19-15:24:17.272526 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00205/att-01/compact.done) = No such file or directory: 5.438 sec on http probe failed: 4, retry 2024/11/19-15:24:17.309055 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-02 2024/11/19-15:24:17.309174 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00200/att-02/compact.done) = No such file or directory: 11.027 sec on http probe failed: 4, retry 2024/11/19-15:24:17.331742 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-00 2024/11/19-15:24:17.331815 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00206/att-00/compact.done) = No such file or directory: 9.211 sec on http probe failed: 4, retry 2024/11/19-15:24:17.452749 139851158521600 [dcompact_etcd.cc:1163] job-00209/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.277 sec, input = 91.966 MiB 2024/11/19-15:24:18.319082 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-00 2024/11/19-15:24:18.319171 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00208/att-00/compact.done) = No such file or directory: 2.252 sec on http probe failed: 1, retry 2024/11/19-15:24:18.331728 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-00 2024/11/19-15:24:18.331793 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00207/att-00/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:24:18.553845 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-01 2024/11/19-15:24:18.553977 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00205/att-01/compact.done) = No such file or directory: 6.722 sec on http probe failed: 5, giveup 2024/11/19-15:24:18.554003 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00205/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 63.647 sec, real 7.012 (submit 281.844 ms wait 6.722 s), input 63.647 MiB, send shutdown 2024/11/19-15:24:18.559202 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-01 2024/11/19-15:24:18.559473 139851137550080 [dcompact_etcd.cc:1773] job-00205/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:18.839494 139851137550080 [dcompact_etcd.cc:1163] job-00205/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 63.647 MiB 2024/11/19-15:24:19.315482 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-02 2024/11/19-15:24:19.315581 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00200/att-02/compact.done) = No such file or directory: 13.036 sec on http probe failed: 5, giveup 2024/11/19-15:24:19.315600 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00200/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.783 sec, real 13.345 (submit 303.217 ms wait 13.036 s), input 253.783 MiB, send shutdown 2024/11/19-15:24:19.324061 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00200/att-02 2024/11/19-15:24:19.324294 139851169007360 [dcompact_etcd.cc:1773] job-00200/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:19.324551 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00200) = success 2024/11/19-15:24:19.324558 139851169007360 [dcompact_etcd.cc:1773] job-00200/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:19.324577 139851169007360 (Original Log Time 2024/11/19-15:23:39.276792) EVENT_LOG_v1 {"time_micros": 1732001019276782, "job": 200, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4611, 4612, 4613, 4620], "score": 2.14854, "input_data_size": 262146390, "oldest_snapshot_seqno": 140361717} 2024/11/19-15:24:19.324583 139851169007360 [compaction_job.cc:2437] [default] [JOB 200] Compacting 4@2 files to L3, score 2.15, subcompactions 1 : 1 2024/11/19-15:24:19.324591 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 198 Base level 2, inputs: [4611(61MB) 4612(62MB) 4613(61MB) 4620(63MB)] 2024/11/19-15:24:19.328201 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005165.sst) /storage/mytopling/..//tmp/Topling-82038-mGBulo 2024/11/19-15:24:19.339584 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-00 2024/11/19-15:24:19.339656 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00206/att-00/compact.done) = No such file or directory: 11.219 sec on http probe failed: 5, giveup 2024/11/19-15:24:19.339673 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00206/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.235 sec, real 11.519 (submit 292.272 ms wait 11.219 s), input 127.235 MiB, send shutdown 2024/11/19-15:24:19.348900 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-00 2024/11/19-15:24:19.350297 139851148035840 [dcompact_etcd.cc:1773] job-00206/att-00: CleanFiles 0.001 sec : 0.000 2024/11/19-15:24:19.513157 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005165.sst): first pass time = 0.18's, 725.751'MB/sec 2024/11/19-15:24:19.513157 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005165.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:24:19.513211 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005165.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0083 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:24:19.523908 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005165.sst): index pass time = 0.01's, 96.803'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 129429 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:24:19.614132 139851148035840 [dcompact_etcd.cc:1163] job-00206/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.264 sec, input = 127.235 MiB 2024/11/19-15:24:19.777983 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-00 2024/11/19-15:24:19.778112 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00209/att-00/compact.done) = No such file or directory: 2.307 sec on http probe failed: 1, retry 2024/11/19-15:24:20.133802 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-00 2024/11/19-15:24:20.133938 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00208/att-00/compact.done) = No such file or directory: 4.065 sec on http probe failed: 2, retry 2024/11/19-15:24:20.344060 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-00 2024/11/19-15:24:20.344155 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00207/att-00/compact.done) = No such file or directory: 5.190 sec on http probe failed: 2, retry 2024/11/19-15:24:20.443483 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-02 2024/11/19-15:24:20.443603 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00205/att-02/compact.done) = No such file or directory: 1.591 sec on http probe failed: 1, retry 2024/11/19-15:24:20.626061 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005165.sst): old prealloc_size = 147639500, real_size = 40353232 2024/11/19-15:24:20.626164 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005165.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129429 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.01 avg-zval = 309.84 2024/11/19-15:24:20.626219 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005165.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 1.113 sec, Key+Value = 0.134 GB 2024/11/19-15:24:20.746545 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005165.sst): second pass time = 1.11's, 118.285'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 346.159'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2579.374'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 0.369'MB/sec write SST data time = 0.03's, 1279.464'MB/sec dict compress time = 0.00's, 1195.374'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 19670.450'MB/sec zip my value time = 1.11's, unzip length = 0.132'GB zip my value throughput = 118.285'MB/sec zip pipeline throughput = 118.285'MB/sec entries = 129429 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.01 avg-zval = 309.84 usrkeys = 129429 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.01 avg-zval = 309.84 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0980 value = 3.2824 dict = 1.00 all = 3.2186 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 12.577515 GB avg = 0.255 KB (by entry num) total key len = 1.274478 GB avg = 0.026 KB total ukey len = 0.879723 GB avg = 0.018 KB total ukey num = 0.049344326 Billion total entry num = 0.049344326 Billion write speed all = 40.293379858 MB/sec (with seq num) write speed all = 39.145097589 MB/sec (without seq num) 2024/11/19-15:24:20.747786 139851169007360 [compaction_job.cc:1948] [default] [JOB 200] Generated table #5165: 129429 keys, 41700401 bytes, temperature: kUnknown 2024/11/19-15:24:20.747844 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001060747817, "cf_name": "default", "job": 200, "event": "table_file_creation", "file_number": 5165, "file_size": 41700401, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 40101888, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588580, "raw_average_key_size": 20, "raw_value_size": 131630167, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129429, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001059, "oldest_key_time": 0, "file_creation_time": 1732001059, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5165, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:20.768141 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005166.sst) /storage/mytopling/..//tmp/Topling-82038-oWz6zq 2024/11/19-15:24:20.979848 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005166.sst): first pass time = 0.21's, 623.047'MB/sec 2024/11/19-15:24:20.979853 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005166.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.132 GB 2024/11/19-15:24:20.979898 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005166.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0081 GB, waited 0.000 sec, Key+Value = 0.132 GB 2024/11/19-15:24:20.988348 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005166.sst): index pass time = 0.01's, 120.434'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 127188 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:24:21.316836 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-01 2024/11/19-15:24:21.316960 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00204/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:24:21.628112 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-00 2024/11/19-15:24:21.628208 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00209/att-00/compact.done) = No such file or directory: 4.165 sec on http probe failed: 2, retry 2024/11/19-15:24:21.744515 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-02 2024/11/19-15:24:21.744614 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00205/att-02/compact.done) = No such file or directory: 2.877 sec on http probe failed: 2, retry 2024/11/19-15:24:21.847663 139852349699840 [db_impl_write.cc:2181] [default] New memtable created with log file: #5167. Immutable memtables: 0. 2024/11/19-15:24:21.851910 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 210] Syncing log #5058 2024/11/19-15:24:21.852040 139850957195008 (Original Log Time 2024/11/19-15:24:21.848334) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:24:21.852056 139850957195008 [flush_job.cc:879] [default] [JOB 210] Flushing memtable with next log file: 5167 2024/11/19-15:24:21.852087 139850957195008 EVENT_LOG_v1 {"time_micros": 1732001061852076, "job": 210, "event": "flush_started", "num_memtables": 1, "num_entries": 1415092, "num_deletes": 0, "total_data_size": 500650547, "memory_usage": 545521032, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:24:21.852094 139850957195008 [flush_job.cc:910] [default] [JOB 210] Level-0 flush table #5168: started 2024/11/19-15:24:21.853108 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000026.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/005168.sst in ConvertToSST, mem_size = 520.249 M, time = 995.008 us 2024/11/19-15:24:21.946753 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-00 2024/11/19-15:24:21.946907 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00208/att-00/compact.done) = No such file or directory: 5.877 sec on http probe failed: 3, retry 2024/11/19-15:24:22.110879 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005166.sst): old prealloc_size = 147639500, real_size = 39652896 2024/11/19-15:24:22.111084 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005166.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 127188 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1293 GB zip-val = 0.0394 GB avg-val =1016.99 avg-zval = 309.82 2024/11/19-15:24:22.111184 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005166.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 1.131 sec, Key+Value = 0.132 GB 2024/11/19-15:24:22.152488 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/005168.sst): fsize = 520.251 M, time(ms): open: 0.047, seek: 0.018, finish: 0.059, meta: 0.017, Flush: 0.022, sync: 299.181, close: 0.009, all: 299.353 2024/11/19-15:24:22.153581 139850957195008 [flush_job.cc:1083] [default] [JOB 210] Flush lasted 301544 microseconds, and 224307 cpu microseconds. 2024/11/19-15:24:22.175452 139850957195008 (Original Log Time 2024/11/19-15:24:22.152533) [flush_job.cc:1029] [default] [JOB 210] Level-0 flush table #5168: 545522458 bytes OK (needs compaction) 2024/11/19-15:24:22.175463 139850957195008 (Original Log Time 2024/11/19-15:24:22.153598) [memtable_list.cc:560] [default] Level-0 commit table #5168 started 2024/11/19-15:24:22.175466 139850957195008 (Original Log Time 2024/11/19-15:24:22.175380) [memtable_list.cc:759] [default] Level-0 commit table #5168: memtable #1 done 2024/11/19-15:24:22.175469 139850957195008 (Original Log Time 2024/11/19-15:24:22.175405) EVENT_LOG_v1 {"time_micros": 1732001062175399, "job": 210, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [2, 140, 57, 51, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:24:22.175473 139850957195008 (Original Log Time 2024/11/19-15:24:22.175425) [db_impl_compaction_flush.cc:310] [default] Level summary: files[2 140 57 51 0 0 0] max score 7.40 (171 files need compaction) 2024/11/19-15:24:22.244541 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005166.sst): second pass time = 1.13's, 114.374'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.13's, 306.578'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1321.761'MB/sec (index lex order gen) rebuild zvType time = 0.10's, 0.325'MB/sec write SST data time = 0.04's, 1152.574'MB/sec dict compress time = 0.00's, 1766.392'MB/sec z-dict build time = 0.00's, sample length = 1.355'MB, throughput = 22153.827'MB/sec zip my value time = 1.13's, unzip length = 0.129'GB zip my value throughput = 114.374'MB/sec zip pipeline throughput = 114.374'MB/sec entries = 127188 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.82 usrkeys = 127188 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.82 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0015 GB value = 0.1293 GB dict = 1.36 MB all = 0.1319 GB } __ZipSize{ index = 0.0002 GB value = 0.0394 GB dict = 1.36 MB all = 0.0410 GB } UnZip/Zip{ index = 7.0975 value = 3.2825 dict = 1.00 all = 3.2186 } Zip/UnZip{ index = 0.1409 value = 0.3046 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 12.706863 GB avg = 0.257 KB (by entry num) total key len = 1.277022 GB avg = 0.026 KB total ukey len = 0.881249 GB avg = 0.018 KB total ukey num = 0.049471514 Billion total entry num = 0.049471514 Billion write speed all = 40.500555567 MB/sec (with seq num) write speed all = 39.354308244 MB/sec (without seq num) 2024/11/19-15:24:22.245874 139851169007360 [compaction_job.cc:1948] [default] [JOB 200] Generated table #5166: 127188 keys, 40977830 bytes, temperature: kUnknown 2024/11/19-15:24:22.245933 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001062245901, "cf_name": "default", "job": 200, "event": "table_file_creation", "file_number": 5166, "file_size": 40977830, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 39405904, "index_size": 215040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2543760, "raw_average_key_size": 20, "raw_value_size": 129348455, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 127188, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001060, "oldest_key_time": 0, "file_creation_time": 1732001060, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5166, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:22.265976 139851169007360 [compaction_job.cc:703] job-00200: subcompact[0], size: 0.082678 G, files: /storage/mytopling/.rocksdb [005165,005166] 2024/11/19-15:24:22.267849 139851169007360 (Original Log Time 2024/11/19-15:24:19.324620) EVENT_LOG_v1 {"time_micros": 1732001059324604, "job": 200, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4611, 4612, 4613, 4620], "score": 2.14854, "input_data_size": 262146390, "oldest_snapshot_seqno": 140361717} 2024/11/19-15:24:22.267860 139851169007360 (Original Log Time 2024/11/19-15:24:22.267366) [compaction_job.cc:2022] [default] [JOB 200] Compacted 4@2 files to L3 => 82678231 bytes 2024/11/19-15:24:22.267864 139851169007360 (Original Log Time 2024/11/19-15:24:22.267699) [compaction_job.cc:1224] [default] compacted to: files[2 140 53 53 0 0 0] max score 7.40 (171 files need compaction), MB/sec: 89.1 rd, 28.1 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(250.0, 0.0 +0.0 blob) out(78.8 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 256617, records dropped: 0 output_compression: Snappy 2024/11/19-15:24:22.267878 139851169007360 (Original Log Time 2024/11/19-15:24:22.267725) EVENT_LOG_v1 {"time_micros": 1732001062267712, "cf": "default", "job": 200, "event": "compaction_finished", "compaction_time_micros": 2941370, "compaction_time_cpu_micros": 1153354, "output_level": 3, "num_output_files": 2, "total_output_size": 82678231, "num_input_records": 256617, "num_output_records": 256617, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 140, 53, 53, 0, 0, 0]} 2024/11/19-15:24:22.268826 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004885.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:22.268848 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001062268845, "job": 200, "event": "table_file_deletion", "file_number": 4885} 2024/11/19-15:24:22.269809 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004887.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:22.269824 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001062269822, "job": 200, "event": "table_file_deletion", "file_number": 4887} 2024/11/19-15:24:22.270734 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004888.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:22.270748 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001062270747, "job": 200, "event": "table_file_deletion", "file_number": 4888} 2024/11/19-15:24:22.271639 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004889.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:22.271652 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001062271651, "job": 200, "event": "table_file_deletion", "file_number": 4889} 2024/11/19-15:24:22.281309 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004953.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:22.281330 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001062281328, "job": 200, "event": "table_file_deletion", "file_number": 4953} 2024/11/19-15:24:22.290934 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004954.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:22.290955 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001062290953, "job": 200, "event": "table_file_deletion", "file_number": 4954} 2024/11/19-15:24:22.300394 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004956.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:22.300415 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001062300413, "job": 200, "event": "table_file_deletion", "file_number": 4956} 2024/11/19-15:24:22.309228 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004959.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:22.309249 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001062309247, "job": 200, "event": "table_file_deletion", "file_number": 4959} 2024/11/19-15:24:22.316112 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004968.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:22.316131 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001062316129, "job": 200, "event": "table_file_deletion", "file_number": 4968} 2024/11/19-15:24:22.322930 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004969.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:22.322950 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001062322948, "job": 200, "event": "table_file_deletion", "file_number": 4969} 2024/11/19-15:24:22.329638 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004970.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:22.329657 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001062329655, "job": 200, "event": "table_file_deletion", "file_number": 4970} 2024/11/19-15:24:22.336507 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004971.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:22.336526 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001062336524, "job": 200, "event": "table_file_deletion", "file_number": 4971} 2024/11/19-15:24:22.336696 139851169007360 [compaction_job.cc:2437] [default] [JOB 211] Compacting 4@1 files to L2, score 7.40, subcompactions 1 : 1 2024/11/19-15:24:22.336704 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 209 Base level 1, inputs: [5054(31MB) 5063(31MB) 5067(31MB) 5072(31MB)] 2024/11/19-15:24:22.352653 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-00 2024/11/19-15:24:22.352721 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00207/att-00/compact.done) = No such file or directory: 7.202 sec on http probe failed: 3, retry 2024/11/19-15:24:22.656426 139851169007360 [dcompact_etcd.cc:1163] job-00211/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.319 sec, input = 127.817 MiB 2024/11/19-15:24:22.801382 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-01 2024/11/19-15:24:22.801480 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00206/att-01/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:24:23.025771 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-02 2024/11/19-15:24:23.025877 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00205/att-02/compact.done) = No such file or directory: 4.178 sec on http probe failed: 3, retry 2024/11/19-15:24:23.326361 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-01 2024/11/19-15:24:23.326483 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00204/att-01/compact.done) = No such file or directory: 7.013 sec on http probe failed: 2, retry 2024/11/19-15:24:23.475777 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-00 2024/11/19-15:24:23.475876 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00209/att-00/compact.done) = No such file or directory: 6.015 sec on http probe failed: 3, retry 2024/11/19-15:24:23.757417 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-00 2024/11/19-15:24:23.757497 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00208/att-00/compact.done) = No such file or directory: 7.690 sec on http probe failed: 4, retry 2024/11/19-15:24:24.308085 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-02 2024/11/19-15:24:24.308189 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00205/att-02/compact.done) = No such file or directory: 5.459 sec on http probe failed: 4, retry 2024/11/19-15:24:24.360811 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-00 2024/11/19-15:24:24.360926 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00207/att-00/compact.done) = No such file or directory: 9.210 sec on http probe failed: 4, retry 2024/11/19-15:24:24.811217 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-01 2024/11/19-15:24:24.811299 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00206/att-01/compact.done) = No such file or directory: 5.187 sec on http probe failed: 2, retry 2024/11/19-15:24:25.322984 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-00 2024/11/19-15:24:25.323097 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00209/att-00/compact.done) = No such file or directory: 7.863 sec on http probe failed: 4, retry 2024/11/19-15:24:25.334144 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-01 2024/11/19-15:24:25.334222 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00204/att-01/compact.done) = No such file or directory: 9.023 sec on http probe failed: 3, retry 2024/11/19-15:24:25.566812 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-00 2024/11/19-15:24:25.566912 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00208/att-00/compact.done) = No such file or directory: 9.500 sec on http probe failed: 5, giveup 2024/11/19-15:24:25.566937 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00208/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 90.065 sec, real 9.826 (submit 317.580 ms wait 9.500 s), input 90.065 MiB, send shutdown 2024/11/19-15:24:25.574357 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-00 2024/11/19-15:24:25.574580 139850978166528 [dcompact_etcd.cc:1773] job-00208/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:25.590004 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-02 2024/11/19-15:24:25.590068 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00205/att-02/compact.done) = No such file or directory: 6.742 sec on http probe failed: 5, giveup 2024/11/19-15:24:25.590085 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00205/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 63.647 sec, real 7.031 (submit 279.908 ms wait 6.742 s), input 63.647 MiB, send shutdown 2024/11/19-15:24:25.594792 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00205/att-02 2024/11/19-15:24:25.594980 139851137550080 [dcompact_etcd.cc:1773] job-00205/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:25.595208 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00205) = success 2024/11/19-15:24:25.595213 139851137550080 [dcompact_etcd.cc:1773] job-00205/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:25.595237 139851137550080 (Original Log Time 2024/11/19-15:24:04.393986) EVENT_LOG_v1 {"time_micros": 1732001044393977, "job": 205, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5015, 5018, 5019, 5021], "score": 8.38329, "input_data_size": 67053867, "oldest_snapshot_seqno": 141594929} 2024/11/19-15:24:25.595245 139851137550080 [compaction_job.cc:2437] [default] [JOB 205] Compacting 4@1 files to L2, score 8.38, subcompactions 1 : 1 2024/11/19-15:24:25.595253 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 203 Base level 1, inputs: [5015(6963B) 5018(32KB) 5019(31MB) 5021(31MB)] 2024/11/19-15:24:25.595461 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005169.sst) /storage/mytopling/..//tmp/Topling-82038-qEtT8m 2024/11/19-15:24:25.595567 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005169.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:24:25.595645 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005169.sst): index pass time = 0.00's, 1.369'MB/sec index type = UintIndex_AllOne usrkeys = 20 min-keylen = 4 max-keylen = 4 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 8.00 2024/11/19-15:24:25.595681 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005169.sst): first pass time = 0.00's, 12.446'MB/sec 2024/11/19-15:24:25.595708 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005169.sst): old prealloc_size = 73759253, real_size = 2912 2024/11/19-15:24:25.595722 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005169.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 20 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 8.00 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 115.00 avg-zval = 127.20 2024/11/19-15:24:25.595935 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005169.sst): second pass time = 0.00's,29870.130'MB/sec, value only(87.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 19.788'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 12.183'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.176'MB/sec write SST data time = 0.00's, 25.209'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 29870.130'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 20 avg-key = 16.00 avg-zkey = 9.60 avg-val = 115.00 avg-zval = 127.20 usrkeys = 20 avg-key = 8.00 avg-zkey = 9.60 avg-val = 122.00 avg-zval = 127.20 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 20 bytes = 80 } seq expand size = 140 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.8333 value = 0.9041 dict = 0.00 all = 0.6598 } Zip/UnZip{ index = 1.2000 value = 1.1061 dict = 0.00 all = 1.5156 } ---------------------------- total value len = 12.706866 GB avg = 0.257 KB (by entry num) total key len = 1.277022 GB avg = 0.026 KB total ukey len = 0.881250 GB avg = 0.018 KB total ukey num = 0.049471534 Billion total entry num = 0.049471534 Billion write speed all = 40.111227493 MB/sec (with seq num) write speed all = 38.975998704 MB/sec (without seq num) 2024/11/19-15:24:25.596112 139851137550080 [compaction_job.cc:1948] [default] [JOB 205] Generated table #5169: 20 keys, 3971 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:25.596145 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001065596125, "cf_name": "default", "job": 205, "event": "table_file_creation", "file_number": 5169, "file_size": 3971, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130572162, "largest_seqno": 139714809, "table_properties": {"data_size": 2544, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 320, "raw_average_key_size": 16, "raw_value_size": 2300, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 20, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001065, "oldest_key_time": 0, "file_creation_time": 1732001065, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5169, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:25.596308 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005170.sst) /storage/mytopling/..//tmp/Topling-82038-txv8q3 2024/11/19-15:24:25.596466 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005170.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:24:25.596603 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005170.sst): first pass time = 0.00's, 94.412'MB/sec 2024/11/19-15:24:25.596617 139834066732800 [fixed_len_key_index.cc:363] FixedLenKeyIndex build_cache Finish: num_keys = 193, fixlen = 8, pref_len = 4, cplen = 3, time = 0.000 sec 2024/11/19-15:24:25.596678 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005170.sst): index pass time = 0.00's, 7.951'MB/sec index type = FixedLenKeyIndex usrkeys = 193 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 54.38 2024/11/19-15:24:25.596715 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005170.sst): old prealloc_size = 73759253, real_size = 34672 2024/11/19-15:24:25.596730 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005170.sst): index type = FixedLenKeyIndex , store type = PlainBlobStore usrkeys = 193 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 54.38 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 121.17 avg-zval = 124.19 2024/11/19-15:24:25.596982 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005170.sst): second pass time = 0.00's,343911.765'MB/sec, value only(85.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 141.986'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 666.794'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 1.237'MB/sec write SST data time = 0.00's, 179.872'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 343911.765'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 193 avg-key = 20.00 avg-zkey = 54.38 avg-val = 121.17 avg-zval = 124.19 usrkeys = 193 avg-key = 12.00 avg-zkey = 54.38 avg-val = 128.17 avg-zval = 124.19 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 193 bytes = 592 } seq expand size = 1351 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.2207 value = 0.9757 dict = 0.00 all = 0.7518 } Zip/UnZip{ index = 4.5320 value = 1.0249 dict = 0.00 all = 1.3301 } ---------------------------- total value len = 12.706890 GB avg = 0.257 KB (by entry num) total key len = 1.277026 GB avg = 0.026 KB total ukey len = 0.881252 GB avg = 0.018 KB total ukey num = 0.049471727 Billion total entry num = 0.049471727 Billion write speed all = 40.111187538 MB/sec (with seq num) write speed all = 38.975957772 MB/sec (without seq num) 2024/11/19-15:24:25.597272 139851137550080 [compaction_job.cc:1948] [default] [JOB 205] Generated table #5170: 193 keys, 36240 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:25.597301 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001065597284, "cf_name": "default", "job": 205, "event": "table_file_creation", "file_number": 5170, "file_size": 36240, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130203043, "largest_seqno": 139804824, "table_properties": {"data_size": 23968, "index_size": 10496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3860, "raw_average_key_size": 20, "raw_value_size": 23386, "raw_average_value_size": 121, "num_data_blocks": 1, "num_entries": 193, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-FixedLenKeyIndex-PlainBlobStore", "creation_time": 1732001065, "oldest_key_time": 0, "file_creation_time": 1732001065, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5170, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:25.597451 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005171.sst) /storage/mytopling/..//tmp/Topling-82038-vdSTQ9 2024/11/19-15:24:25.657194 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005171.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:24:25.662469 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005171.sst): index pass time = 0.01's, 228.190'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 99726 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:24:25.724636 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005171.sst): first pass time = 0.13's, 524.542'MB/sec 2024/11/19-15:24:25.724688 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005171.sst): old prealloc_size = 73759253, real_size = 64833680 2024/11/19-15:24:25.724758 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005171.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 99726 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0643 GB zip-val = 0.0646 GB avg-val = 644.92 avg-zval = 648.17 2024/11/19-15:24:25.724850 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005171.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.068 sec, Key+Value = 0.067 GB 2024/11/19-15:24:25.791473 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005171.sst): second pass time = 0.00's,300537845.794'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 974.617'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2573.084'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.575'MB/sec write SST data time = 0.02's, 2785.140'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.064'GB zip my value throughput = 300537845.794'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 99726 avg-key = 24.00 avg-zkey = 1.70 avg-val = 644.92 avg-zval = 648.17 usrkeys = 99726 avg-key = 16.00 avg-zkey = 1.70 avg-val = 651.92 avg-zval = 648.17 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 99726 bytes = 261792 } seq expand size = 698082 multi value expand size = 0 cnt WriteAppend = 7898 UnZipSize{ index = 0.0016 GB value = 0.0643 GB dict = 0.00 MB all = 0.0667 GB } __ZipSize{ index = 0.0002 GB value = 0.0646 GB dict = 0.00 MB all = 0.0651 GB } UnZip/Zip{ index = 9.4330 value = 0.9950 dict = 0.00 all = 1.0251 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9755 } ---------------------------- total value len = 12.771903 GB avg = 0.258 KB (by entry num) total key len = 1.279419 GB avg = 0.026 KB total ukey len = 0.882848 GB avg = 0.018 KB total ukey num = 0.049571453 Billion total entry num = 0.049571453 Billion write speed all = 40.282064426 MB/sec (with seq num) write speed all = 39.145180442 MB/sec (without seq num) 2024/11/19-15:24:25.795179 139851137550080 [compaction_job.cc:1948] [default] [JOB 205] Generated table #5171: 99726 keys, 65072474 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:25.795240 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001065795210, "cf_name": "default", "job": 205, "event": "table_file_creation", "file_number": 5171, "file_size": 65072474, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130203948, "largest_seqno": 131900829, "table_properties": {"data_size": 64639424, "index_size": 169152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2393424, "raw_average_key_size": 24, "raw_value_size": 64315099, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 99726, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001065, "oldest_key_time": 0, "file_creation_time": 1732001065, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5171, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:25.810022 139851137550080 [compaction_job.cc:703] job-00205: subcompact[0], size: 0.065113 G, files: /storage/mytopling/.rocksdb [005169,005170,005171] 2024/11/19-15:24:25.810972 139851137550080 (Original Log Time 2024/11/19-15:24:25.595281) EVENT_LOG_v1 {"time_micros": 1732001065595266, "job": 205, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5015, 5018, 5019, 5021], "score": 8.38329, "input_data_size": 67053867, "oldest_snapshot_seqno": 141594929} 2024/11/19-15:24:25.810983 139851137550080 (Original Log Time 2024/11/19-15:24:25.810495) [compaction_job.cc:2022] [default] [JOB 205] Compacted 4@1 files to L2 => 65112685 bytes 2024/11/19-15:24:25.810987 139851137550080 (Original Log Time 2024/11/19-15:24:25.810844) [compaction_job.cc:1224] [default] compacted to: files[2 136 56 53 0 0 0] max score 7.15 (170 files need compaction), MB/sec: 312.2 rd, 303.2 wr, level 2, files in(4, 0) out(3 +0 blob) MB in(63.9, 0.0 +0.0 blob) out(62.1 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 99939, records dropped: 0 output_compression: Snappy 2024/11/19-15:24:25.811003 139851137550080 (Original Log Time 2024/11/19-15:24:25.810871) EVENT_LOG_v1 {"time_micros": 1732001065810858, "cf": "default", "job": 205, "event": "compaction_finished", "compaction_time_micros": 214756, "compaction_time_cpu_micros": 212357, "output_level": 2, "num_output_files": 3, "total_output_size": 65112685, "num_input_records": 99939, "num_output_records": 99939, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 136, 56, 53, 0, 0, 0]} 2024/11/19-15:24:25.811081 139851137550080 [compaction_job.cc:2437] [default] [JOB 212] Compacting 4@1 files to L2, score 7.15, subcompactions 1 : 1 2024/11/19-15:24:25.811089 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 210 Base level 1, inputs: [5076(31MB) 5080(31MB) 5084(31MB) 5088(31MB)] 2024/11/19-15:24:25.862587 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-00 2024/11/19-15:24:25.862659 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00211/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:24:25.876666 139850978166528 [dcompact_etcd.cc:1163] job-00208/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.302 sec, input = 90.065 MiB 2024/11/19-15:24:26.120907 139851137550080 [dcompact_etcd.cc:1163] job-00212/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.309 sec, input = 127.823 MiB 2024/11/19-15:24:26.369721 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-00 2024/11/19-15:24:26.369823 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00207/att-00/compact.done) = No such file or directory: 11.219 sec on http probe failed: 5, giveup 2024/11/19-15:24:26.369842 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00207/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.238 sec, real 11.561 (submit 333.030 ms wait 11.219 s), input 127.238 MiB, send shutdown 2024/11/19-15:24:26.377828 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-00 2024/11/19-15:24:26.378074 139851127064320 [dcompact_etcd.cc:1773] job-00207/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:26.649440 139851127064320 [dcompact_etcd.cc:1163] job-00207/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.271 sec, input = 127.238 MiB 2024/11/19-15:24:26.820461 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-01 2024/11/19-15:24:26.820548 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00206/att-01/compact.done) = No such file or directory: 7.197 sec on http probe failed: 3, retry 2024/11/19-15:24:27.171320 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-00 2024/11/19-15:24:27.171418 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00209/att-00/compact.done) = No such file or directory: 9.710 sec on http probe failed: 5, giveup 2024/11/19-15:24:27.171437 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00209/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 91.966 sec, real 9.996 (submit 277.397 ms wait 9.710 s), input 91.966 MiB, send shutdown 2024/11/19-15:24:27.176462 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-00 2024/11/19-15:24:27.176695 139851158521600 [dcompact_etcd.cc:1773] job-00209/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:27.342677 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-01 2024/11/19-15:24:27.342785 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00204/att-01/compact.done) = No such file or directory: 11.030 sec on http probe failed: 4, retry 2024/11/19-15:24:27.610795 139851158521600 [dcompact_etcd.cc:1163] job-00209/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.434 sec, input = 91.966 MiB 2024/11/19-15:24:27.872411 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-00 2024/11/19-15:24:27.872514 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00211/att-00/compact.done) = No such file or directory: 5.206 sec on http probe failed: 2, retry 2024/11/19-15:24:28.137594 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-01 2024/11/19-15:24:28.137685 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00208/att-01/compact.done) = No such file or directory: 2.252 sec on http probe failed: 1, retry 2024/11/19-15:24:28.829091 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-01 2024/11/19-15:24:28.829183 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00206/att-01/compact.done) = No such file or directory: 9.207 sec on http probe failed: 4, retry 2024/11/19-15:24:29.330423 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-00 2024/11/19-15:24:29.330525 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00212/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:24:29.351621 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-01 2024/11/19-15:24:29.351703 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00204/att-01/compact.done) = No such file or directory: 13.039 sec on http probe failed: 5, giveup 2024/11/19-15:24:29.351720 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00204/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.794 sec, real 13.281 (submit 233.513 ms wait 13.039 s), input 253.794 MiB, send shutdown 2024/11/19-15:24:29.360408 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-01 2024/11/19-15:24:29.360615 139851179493120 [dcompact_etcd.cc:1773] job-00204/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:29.649848 139851179493120 [dcompact_etcd.cc:1163] job-00204/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 253.794 MiB 2024/11/19-15:24:29.839896 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-01 2024/11/19-15:24:29.839990 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00207/att-01/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:24:29.880232 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-00 2024/11/19-15:24:29.880299 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00211/att-00/compact.done) = No such file or directory: 7.216 sec on http probe failed: 3, retry 2024/11/19-15:24:29.915813 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-01 2024/11/19-15:24:29.915884 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00209/att-01/compact.done) = No such file or directory: 2.299 sec on http probe failed: 1, retry 2024/11/19-15:24:29.947788 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-01 2024/11/19-15:24:29.947850 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00208/att-01/compact.done) = No such file or directory: 4.062 sec on http probe failed: 2, retry 2024/11/19-15:24:30.838613 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-01 2024/11/19-15:24:30.838692 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00206/att-01/compact.done) = No such file or directory: 11.215 sec on http probe failed: 5, giveup 2024/11/19-15:24:30.838710 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00206/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.235 sec, real 11.488 (submit 263.746 ms wait 11.215 s), input 127.235 MiB, send shutdown 2024/11/19-15:24:30.847553 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-01 2024/11/19-15:24:30.847771 139851148035840 [dcompact_etcd.cc:1773] job-00206/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:31.144431 139851148035840 [dcompact_etcd.cc:1163] job-00206/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.297 sec, input = 127.235 MiB 2024/11/19-15:24:31.335744 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-00 2024/11/19-15:24:31.335818 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00212/att-00/compact.done) = No such file or directory: 5.210 sec on http probe failed: 2, retry 2024/11/19-15:24:31.758180 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-01 2024/11/19-15:24:31.758246 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00208/att-01/compact.done) = No such file or directory: 5.873 sec on http probe failed: 3, retry 2024/11/19-15:24:31.762449 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-01 2024/11/19-15:24:31.762512 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00209/att-01/compact.done) = No such file or directory: 4.144 sec on http probe failed: 2, retry 2024/11/19-15:24:31.847862 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-01 2024/11/19-15:24:31.847929 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00207/att-01/compact.done) = No such file or directory: 5.191 sec on http probe failed: 2, retry 2024/11/19-15:24:31.885304 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-00 2024/11/19-15:24:31.885368 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00211/att-00/compact.done) = No such file or directory: 9.224 sec on http probe failed: 4, retry 2024/11/19-15:24:33.343620 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-00 2024/11/19-15:24:33.343721 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00212/att-00/compact.done) = No such file or directory: 7.215 sec on http probe failed: 3, retry 2024/11/19-15:24:33.564546 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-01 2024/11/19-15:24:33.564627 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00208/att-01/compact.done) = No such file or directory: 7.683 sec on http probe failed: 4, retry 2024/11/19-15:24:33.610260 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-01 2024/11/19-15:24:33.610323 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00209/att-01/compact.done) = No such file or directory: 5.991 sec on http probe failed: 3, retry 2024/11/19-15:24:33.856244 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-01 2024/11/19-15:24:33.856322 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00207/att-01/compact.done) = No such file or directory: 7.199 sec on http probe failed: 3, retry 2024/11/19-15:24:33.890597 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-00 2024/11/19-15:24:33.890657 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00211/att-00/compact.done) = No such file or directory: 11.229 sec on http probe failed: 5, giveup 2024/11/19-15:24:33.890675 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00211/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.817 sec, real 11.554 (submit 319.279 ms wait 11.229 s), input 127.817 MiB, send shutdown 2024/11/19-15:24:33.896364 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-00 2024/11/19-15:24:33.896570 139851169007360 [dcompact_etcd.cc:1773] job-00211/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:34.155696 139851169007360 [dcompact_etcd.cc:1163] job-00211/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.259 sec, input = 127.817 MiB 2024/11/19-15:24:34.334886 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-02 2024/11/19-15:24:34.335008 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00206/att-02/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:24:34.658599 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-02 2024/11/19-15:24:34.658708 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00204/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:24:35.349387 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-00 2024/11/19-15:24:35.349489 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00212/att-00/compact.done) = No such file or directory: 9.223 sec on http probe failed: 4, retry 2024/11/19-15:24:35.371115 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-01 2024/11/19-15:24:35.371204 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00208/att-01/compact.done) = No such file or directory: 9.489 sec on http probe failed: 5, giveup 2024/11/19-15:24:35.371220 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00208/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 90.065 sec, real 9.797 (submit 302.017 ms wait 9.489 s), input 90.065 MiB, send shutdown 2024/11/19-15:24:35.376796 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-01 2024/11/19-15:24:35.377037 139850978166528 [dcompact_etcd.cc:1773] job-00208/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:35.454253 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-01 2024/11/19-15:24:35.454315 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00209/att-01/compact.done) = No such file or directory: 7.839 sec on http probe failed: 4, retry 2024/11/19-15:24:35.670484 139850978166528 [dcompact_etcd.cc:1163] job-00208/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.293 sec, input = 90.065 MiB 2024/11/19-15:24:35.861201 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-01 2024/11/19-15:24:35.861281 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00207/att-01/compact.done) = No such file or directory: 9.207 sec on http probe failed: 4, retry 2024/11/19-15:24:36.343626 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-02 2024/11/19-15:24:36.343725 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00206/att-02/compact.done) = No such file or directory: 5.191 sec on http probe failed: 2, retry 2024/11/19-15:24:36.666916 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-02 2024/11/19-15:24:36.666996 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00204/att-02/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:24:37.302666 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-01 2024/11/19-15:24:37.302755 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00209/att-01/compact.done) = No such file or directory: 9.683 sec on http probe failed: 5, giveup 2024/11/19-15:24:37.302773 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00209/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 91.966 sec, real 10.126 (submit 434.020 ms wait 9.683 s), input 91.966 MiB, send shutdown 2024/11/19-15:24:37.310688 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-01 2024/11/19-15:24:37.310894 139851158521600 [dcompact_etcd.cc:1773] job-00209/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:37.355570 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-00 2024/11/19-15:24:37.355624 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00212/att-00/compact.done) = No such file or directory: 11.229 sec on http probe failed: 5, giveup 2024/11/19-15:24:37.355639 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00212/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.823 sec, real 11.544 (submit 309.366 ms wait 11.229 s), input 127.823 MiB, send shutdown 2024/11/19-15:24:37.356485 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-01 2024/11/19-15:24:37.356564 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00211/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:24:37.363046 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-00 2024/11/19-15:24:37.363239 139851137550080 [dcompact_etcd.cc:1773] job-00212/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:37.603861 139851158521600 [dcompact_etcd.cc:1163] job-00209/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.293 sec, input = 91.966 MiB 2024/11/19-15:24:37.641109 139851137550080 [dcompact_etcd.cc:1163] job-00212/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.278 sec, input = 127.823 MiB 2024/11/19-15:24:37.870153 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-01 2024/11/19-15:24:37.870228 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00207/att-01/compact.done) = No such file or directory: 11.212 sec on http probe failed: 5, giveup 2024/11/19-15:24:37.870246 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00207/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.238 sec, real 11.492 (submit 271.284 ms wait 11.212 s), input 127.238 MiB, send shutdown 2024/11/19-15:24:37.878080 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-01 2024/11/19-15:24:37.878298 139851127064320 [dcompact_etcd.cc:1773] job-00207/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:37.930838 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-02 2024/11/19-15:24:37.930930 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00208/att-02/compact.done) = No such file or directory: 2.252 sec on http probe failed: 1, retry 2024/11/19-15:24:38.161945 139851127064320 [dcompact_etcd.cc:1163] job-00207/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.284 sec, input = 127.238 MiB 2024/11/19-15:24:38.245648 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #5172. Immutable memtables: 0. 2024/11/19-15:24:38.246108 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 213] Syncing log #5167 2024/11/19-15:24:38.246213 139850967680768 (Original Log Time 2024/11/19-15:24:38.245885) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:24:38.246230 139850967680768 [flush_job.cc:879] [default] [JOB 213] Flushing memtable with next log file: 5172 2024/11/19-15:24:38.246258 139850967680768 EVENT_LOG_v1 {"time_micros": 1732001078246250, "job": 213, "event": "flush_started", "num_memtables": 1, "num_entries": 1308611, "num_deletes": 0, "total_data_size": 486489797, "memory_usage": 537656712, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:24:38.246265 139850967680768 [flush_job.cc:910] [default] [JOB 213] Level-0 flush table #5173: started 2024/11/19-15:24:38.246959 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000027.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/005173.sst in ConvertToSST, mem_size = 512.749 M, time = 681.771 us 2024/11/19-15:24:38.351729 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-02 2024/11/19-15:24:38.351863 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00206/att-02/compact.done) = No such file or directory: 7.199 sec on http probe failed: 3, retry 2024/11/19-15:24:38.671875 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-02 2024/11/19-15:24:38.672006 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00204/att-02/compact.done) = No such file or directory: 9.017 sec on http probe failed: 3, retry 2024/11/19-15:24:38.772856 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/005173.sst): fsize = 512.751 M, time(ms): open: 0.054, seek: 0.018, finish: 0.048, meta: 0.014, Flush: 0.014, sync: 525.715, close: 0.009, all: 525.872 2024/11/19-15:24:38.774051 139850967680768 [flush_job.cc:1083] [default] [JOB 213] Flush lasted 527839 microseconds, and 497538 cpu microseconds. 2024/11/19-15:24:38.794536 139850967680768 (Original Log Time 2024/11/19-15:24:38.772902) [flush_job.cc:1029] [default] [JOB 213] Level-0 flush table #5173: 537658138 bytes OK (needs compaction) 2024/11/19-15:24:38.794547 139850967680768 (Original Log Time 2024/11/19-15:24:38.774068) [memtable_list.cc:560] [default] Level-0 commit table #5173 started 2024/11/19-15:24:38.794551 139850967680768 (Original Log Time 2024/11/19-15:24:38.794464) [memtable_list.cc:759] [default] Level-0 commit table #5173: memtable #1 done 2024/11/19-15:24:38.794553 139850967680768 (Original Log Time 2024/11/19-15:24:38.794489) EVENT_LOG_v1 {"time_micros": 1732001078794483, "job": 213, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [3, 136, 56, 53, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:24:38.794557 139850967680768 (Original Log Time 2024/11/19-15:24:38.794509) [db_impl_compaction_flush.cc:310] [default] Level summary: files[3 136 56 53 0 0 0] max score 6.90 (167 files need compaction) 2024/11/19-15:24:39.365929 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-01 2024/11/19-15:24:39.366053 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00211/att-01/compact.done) = No such file or directory: 5.201 sec on http probe failed: 2, retry 2024/11/19-15:24:39.741011 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-02 2024/11/19-15:24:39.741081 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00208/att-02/compact.done) = No such file or directory: 4.062 sec on http probe failed: 2, retry 2024/11/19-15:24:39.909497 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-02 2024/11/19-15:24:39.909588 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00209/att-02/compact.done) = No such file or directory: 2.299 sec on http probe failed: 1, retry 2024/11/19-15:24:40.359774 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-02 2024/11/19-15:24:40.359877 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00206/att-02/compact.done) = No such file or directory: 9.208 sec on http probe failed: 4, retry 2024/11/19-15:24:40.679311 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-02 2024/11/19-15:24:40.679388 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00204/att-02/compact.done) = No such file or directory: 11.022 sec on http probe failed: 4, retry 2024/11/19-15:24:40.845590 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-01 2024/11/19-15:24:40.845682 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00212/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:24:41.352749 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-02 2024/11/19-15:24:41.352859 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00207/att-02/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:24:41.374151 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-01 2024/11/19-15:24:41.374236 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00211/att-01/compact.done) = No such file or directory: 7.210 sec on http probe failed: 3, retry 2024/11/19-15:24:41.552524 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-02 2024/11/19-15:24:41.552608 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00208/att-02/compact.done) = No such file or directory: 5.872 sec on http probe failed: 3, retry 2024/11/19-15:24:41.753558 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-02 2024/11/19-15:24:41.753653 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00209/att-02/compact.done) = No such file or directory: 4.145 sec on http probe failed: 2, retry 2024/11/19-15:24:42.368260 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-02 2024/11/19-15:24:42.368353 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00206/att-02/compact.done) = No such file or directory: 11.216 sec on http probe failed: 5, giveup 2024/11/19-15:24:42.368372 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00206/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.235 sec, real 11.521 (submit 296.580 ms wait 11.216 s), input 127.235 MiB, send shutdown 2024/11/19-15:24:42.372971 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00206/att-02 2024/11/19-15:24:42.373181 139851148035840 [dcompact_etcd.cc:1773] job-00206/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:42.373405 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00206) = success 2024/11/19-15:24:42.373410 139851148035840 [dcompact_etcd.cc:1773] job-00206/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:42.373427 139851148035840 (Original Log Time 2024/11/19-15:24:07.820384) EVENT_LOG_v1 {"time_micros": 1732001047820376, "job": 206, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5023, 5026, 5030, 5032], "score": 8.25839, "input_data_size": 134026368, "oldest_snapshot_seqno": -1} 2024/11/19-15:24:42.373435 139851148035840 [compaction_job.cc:2437] [default] [JOB 206] Compacting 4@1 files to L2, score 8.26, subcompactions 1 : 1 2024/11/19-15:24:42.373442 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 204 Base level 1, inputs: [5023(31MB) 5026(31MB) 5030(31MB) 5032(31MB)] 2024/11/19-15:24:42.373747 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005174.sst) /storage/mytopling/..//tmp/Topling-82038-6ro1on 2024/11/19-15:24:42.435870 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005174.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:24:42.441158 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005174.sst): index pass time = 0.01's, 229.131'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 100319 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:24:42.505415 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005174.sst): first pass time = 0.13's, 509.733'MB/sec 2024/11/19-15:24:42.505466 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005174.sst): old prealloc_size = 73819750, real_size = 65222912 2024/11/19-15:24:42.505537 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005174.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 100319 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0650 GB avg-val = 644.96 avg-zval = 648.21 2024/11/19-15:24:42.505588 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005174.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.070 sec, Key+Value = 0.067 GB 2024/11/19-15:24:42.575405 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005174.sst): second pass time = 0.00's,288845160.714'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 932.478'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2480.490'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.548'MB/sec write SST data time = 0.02's, 2707.724'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 288845160.714'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 100319 avg-key = 24.00 avg-zkey = 1.70 avg-val = 644.96 avg-zval = 648.21 usrkeys = 100319 avg-key = 16.00 avg-zkey = 1.70 avg-val = 644.96 avg-zval = 648.21 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7942 UnZipSize{ index = 0.0016 GB value = 0.0647 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0002 GB value = 0.0650 GB dict = 0.00 MB all = 0.0652 GB } UnZip/Zip{ index = 9.4356 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 12.836605 GB avg = 0.258 KB (by entry num) total key len = 1.281827 GB avg = 0.026 KB total ukey len = 0.884453 GB avg = 0.018 KB total ukey num = 0.049671772 Billion total entry num = 0.049671772 Billion write speed all = 38.616390527 MB/sec (with seq num) write speed all = 37.529502354 MB/sec (without seq num) 2024/11/19-15:24:42.578954 139851148035840 [compaction_job.cc:1948] [default] [JOB 206] Generated table #5174: 100319 keys, 65199757 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:42.579005 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001082578981, "cf_name": "default", "job": 206, "event": "table_file_creation", "file_number": 5174, "file_size": 65199757, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 65027568, "index_size": 170112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2407656, "raw_average_key_size": 24, "raw_value_size": 64701316, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 100319, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001082, "oldest_key_time": 0, "file_creation_time": 1732001082, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5174, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:42.594390 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005175.sst) /storage/mytopling/..//tmp/Topling-82038-sWduYc 2024/11/19-15:24:42.651691 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005175.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:24:42.656978 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005175.sst): index pass time = 0.01's, 226.378'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 99080 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:24:42.684861 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-02 2024/11/19-15:24:42.685013 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00204/att-02/compact.done) = No such file or directory: 13.030 sec on http probe failed: 5, giveup 2024/11/19-15:24:42.685042 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00204/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.794 sec, real 13.324 (submit 289.162 ms wait 13.030 s), input 253.794 MiB, send shutdown 2024/11/19-15:24:42.690631 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00204/att-02 2024/11/19-15:24:42.690941 139851179493120 [dcompact_etcd.cc:1773] job-00204/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:42.691222 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00204) = success 2024/11/19-15:24:42.691234 139851179493120 [dcompact_etcd.cc:1773] job-00204/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:42.691262 139851179493120 (Original Log Time 2024/11/19-15:24:02.724512) EVENT_LOG_v1 {"time_micros": 1732001042724503, "job": 204, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4623, 4624, 4625, 4783], "score": 2.02504, "input_data_size": 262047388, "oldest_snapshot_seqno": 141581927} 2024/11/19-15:24:42.691274 139851179493120 [compaction_job.cc:2437] [default] [JOB 204] Compacting 4@2 files to L3, score 2.03, subcompactions 1 : 1 2024/11/19-15:24:42.691287 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 202 Base level 2, inputs: [4623(61MB) 4624(62MB) 4625(62MB) 4783(62MB)] 2024/11/19-15:24:42.694202 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005176.sst) /storage/mytopling/..//tmp/Topling-82038-7D72A2 2024/11/19-15:24:42.726825 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005175.sst): first pass time = 0.13's, 500.733'MB/sec 2024/11/19-15:24:42.726879 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005175.sst): old prealloc_size = 73819750, real_size = 64444016 2024/11/19-15:24:42.726972 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005175.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 99080 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0639 GB zip-val = 0.0643 GB avg-val = 645.22 avg-zval = 648.48 2024/11/19-15:24:42.727032 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005175.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.075 sec, Key+Value = 0.066 GB 2024/11/19-15:24:42.816469 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005175.sst): second pass time = 0.00's,215248090.909'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 719.335'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1790.577'MB/sec (index lex order gen) rebuild zvType time = 0.06's, 0.429'MB/sec write SST data time = 0.03's, 2032.404'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.064'GB zip my value throughput = 215248090.909'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 99080 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.22 avg-zval = 648.48 usrkeys = 99080 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.22 avg-zval = 648.48 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7848 UnZipSize{ index = 0.0016 GB value = 0.0639 GB dict = 0.00 MB all = 0.0663 GB } __ZipSize{ index = 0.0002 GB value = 0.0643 GB dict = 0.00 MB all = 0.0644 GB } UnZip/Zip{ index = 9.4290 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1061 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 12.900533 GB avg = 0.259 KB (by entry num) total key len = 1.284205 GB avg = 0.026 KB total ukey len = 0.886038 GB avg = 0.018 KB total ukey num = 0.049770852 Billion total entry num = 0.049770852 Billion write speed all = 38.772188039 MB/sec (with seq num) write speed all = 37.683849404 MB/sec (without seq num) 2024/11/19-15:24:42.820629 139851148035840 [compaction_job.cc:1948] [default] [JOB 206] Generated table #5175: 99080 keys, 64421133 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:42.820695 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001082820662, "cf_name": "default", "job": 206, "event": "table_file_creation", "file_number": 5175, "file_size": 64421133, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64250912, "index_size": 168128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2377920, "raw_average_key_size": 24, "raw_value_size": 63928683, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 99080, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001082, "oldest_key_time": 0, "file_creation_time": 1732001082, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5175, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:42.839606 139851148035840 [compaction_job.cc:703] job-00206: subcompact[0], size: 0.129621 G, files: /storage/mytopling/.rocksdb [005174,005175] 2024/11/19-15:24:42.840591 139851148035840 (Original Log Time 2024/11/19-15:24:42.373469) EVENT_LOG_v1 {"time_micros": 1732001082373454, "job": 206, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5023, 5026, 5030, 5032], "score": 8.25839, "input_data_size": 134026368, "oldest_snapshot_seqno": -1} 2024/11/19-15:24:42.840602 139851148035840 (Original Log Time 2024/11/19-15:24:42.840117) [compaction_job.cc:2022] [default] [JOB 206] Compacted 4@1 files to L2 => 129620890 bytes 2024/11/19-15:24:42.840606 139851148035840 (Original Log Time 2024/11/19-15:24:42.840452) [compaction_job.cc:1224] [default] compacted to: files[3 132 58 53 0 0 0] max score 6.90 (169 files need compaction), MB/sec: 287.5 rd, 278.1 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.8, 0.0 +0.0 blob) out(123.6 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 199399, records dropped: 0 output_compression: Snappy 2024/11/19-15:24:42.840624 139851148035840 (Original Log Time 2024/11/19-15:24:42.840478) EVENT_LOG_v1 {"time_micros": 1732001082840466, "cf": "default", "job": 206, "event": "compaction_finished", "compaction_time_micros": 466155, "compaction_time_cpu_micros": 462436, "output_level": 2, "num_output_files": 2, "total_output_size": 129620890, "num_input_records": 199399, "num_output_records": 199399, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 132, 58, 53, 0, 0, 0]} 2024/11/19-15:24:42.840697 139851148035840 [compaction_job.cc:2437] [default] [JOB 214] Compacting 4@1 files to L2, score 6.90, subcompactions 1 : 1 2024/11/19-15:24:42.840706 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 212 Base level 1, inputs: [5091(31MB) 5095(31MB) 5098(31MB) 5100(31MB)] 2024/11/19-15:24:42.853675 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-01 2024/11/19-15:24:42.853767 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00212/att-01/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:24:42.916709 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005176.sst): first pass time = 0.22's, 603.271'MB/sec 2024/11/19-15:24:42.916713 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005176.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:24:42.916749 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005176.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0083 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:24:42.927628 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005176.sst): index pass time = 0.01's, 95.130'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 129429 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:24:43.146575 139851148035840 [dcompact_etcd.cc:1163] job-00214/att-00: init time wrt = 0.001, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.305 sec, input = 127.822 MiB 2024/11/19-15:24:43.366491 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-02 2024/11/19-15:24:43.366586 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00208/att-02/compact.done) = No such file or directory: 7.684 sec on http probe failed: 4, retry 2024/11/19-15:24:43.367184 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-02 2024/11/19-15:24:43.367252 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00207/att-02/compact.done) = No such file or directory: 5.193 sec on http probe failed: 2, retry 2024/11/19-15:24:43.386444 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-01 2024/11/19-15:24:43.386518 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00211/att-01/compact.done) = No such file or directory: 9.221 sec on http probe failed: 4, retry 2024/11/19-15:24:43.609046 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-02 2024/11/19-15:24:43.609147 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00209/att-02/compact.done) = No such file or directory: 5.993 sec on http probe failed: 3, retry 2024/11/19-15:24:43.917851 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005176.sst): old prealloc_size = 147639500, real_size = 40352192 2024/11/19-15:24:43.917971 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005176.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129429 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.00 avg-zval = 309.83 2024/11/19-15:24:43.918039 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005176.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 1.001 sec, Key+Value = 0.134 GB 2024/11/19-15:24:44.033743 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005176.sst): second pass time = 1.00's, 131.488'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 359.879'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2245.902'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 0.380'MB/sec write SST data time = 0.03's, 1360.982'MB/sec dict compress time = 0.00's, 1940.660'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 21187.674'MB/sec zip my value time = 1.00's, unzip length = 0.132'GB zip my value throughput = 131.488'MB/sec zip pipeline throughput = 131.488'MB/sec entries = 129429 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.00 avg-zval = 309.83 usrkeys = 129429 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.00 avg-zval = 309.83 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0980 value = 3.2825 dict = 1.00 all = 3.2187 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 13.032163 GB avg = 0.261 KB (by entry num) total key len = 1.286793 GB avg = 0.026 KB total ukey len = 0.887591 GB avg = 0.018 KB total ukey num = 0.049900281 Billion total entry num = 0.049900281 Billion write speed all = 39.009259221 MB/sec (with seq num) write speed all = 37.921708983 MB/sec (without seq num) 2024/11/19-15:24:44.034845 139851179493120 [compaction_job.cc:1948] [default] [JOB 204] Generated table #5176: 129429 keys, 41699758 bytes, temperature: kUnknown 2024/11/19-15:24:44.034907 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001084034881, "cf_name": "default", "job": 204, "event": "table_file_creation", "file_number": 5176, "file_size": 41699758, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 40100848, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588580, "raw_average_key_size": 20, "raw_value_size": 131629148, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129429, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001082, "oldest_key_time": 0, "file_creation_time": 1732001082, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5176, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:44.055474 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005177.sst) /storage/mytopling/..//tmp/Topling-82038-fkNjHP 2024/11/19-15:24:44.235650 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005177.sst): first pass time = 0.18's, 732.155'MB/sec 2024/11/19-15:24:44.235658 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005177.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.132 GB 2024/11/19-15:24:44.235695 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005177.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0081 GB, waited 0.000 sec, Key+Value = 0.132 GB 2024/11/19-15:24:44.245613 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005177.sst): index pass time = 0.01's, 102.596'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 127197 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:24:44.863830 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-01 2024/11/19-15:24:44.863934 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00212/att-01/compact.done) = No such file or directory: 7.213 sec on http probe failed: 3, retry 2024/11/19-15:24:45.175962 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-02 2024/11/19-15:24:45.176050 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00208/att-02/compact.done) = No such file or directory: 9.497 sec on http probe failed: 5, giveup 2024/11/19-15:24:45.176068 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00208/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 90.065 sec, real 9.799 (submit 293.360 ms wait 9.497 s), input 90.065 MiB, send shutdown 2024/11/19-15:24:45.180420 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00208/att-02 2024/11/19-15:24:45.180661 139850978166528 [dcompact_etcd.cc:1773] job-00208/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:45.180819 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00208) = success 2024/11/19-15:24:45.180827 139850978166528 [dcompact_etcd.cc:1773] job-00208/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:45.180849 139850978166528 (Original Log Time 2024/11/19-15:24:15.740776) EVENT_LOG_v1 {"time_micros": 1732001055740769, "job": 208, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5044, 5047, 5049, 5016], "score": 7.7591, "input_data_size": 97407686, "oldest_snapshot_seqno": 142380812} 2024/11/19-15:24:45.180865 139850978166528 [compaction_job.cc:2437] [default] [JOB 208] Compacting 4@1 files to L2, score 7.76, subcompactions 1 : 1 2024/11/19-15:24:45.180876 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 206 Base level 1, inputs: [5044(31MB) 5047(17MB) 5049(21MB) 5016(21MB)] 2024/11/19-15:24:45.181160 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005178.sst) /storage/mytopling/..//tmp/Topling-82038-IKwyxK 2024/11/19-15:24:45.231121 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005177.sst): old prealloc_size = 147639500, real_size = 39659760 2024/11/19-15:24:45.231223 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005177.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 127197 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1294 GB zip-val = 0.0394 GB avg-val =1017.01 avg-zval = 309.86 2024/11/19-15:24:45.231280 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005177.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.996 sec, Key+Value = 0.132 GB 2024/11/19-15:24:45.240671 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005178.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.052 GB 2024/11/19-15:24:45.247376 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005178.sst): index pass time = 0.01's, 139.150'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 77431 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0009 GB zip-key = 0.0001 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:24:45.294405 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005178.sst): first pass time = 0.11's, 457.591'MB/sec 2024/11/19-15:24:45.294457 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005178.sst): old prealloc_size = 73819750, real_size = 50357568 2024/11/19-15:24:45.294516 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005178.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 77431 prefix = 4 raw-key = 0.0009 GB zip-key = 0.0001 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0500 GB zip-val = 0.0502 GB avg-val = 645.15 avg-zval = 648.40 2024/11/19-15:24:45.294612 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005178.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.054 sec, Key+Value = 0.052 GB 2024/11/19-15:24:45.347957 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005178.sst): second pass time = 0.00's,217193256.522'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 945.148'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2303.141'MB/sec (index lex order gen) rebuild zvType time = 0.03's, 0.559'MB/sec write SST data time = 0.02's, 2692.831'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.050'GB zip my value throughput = 217193256.522'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 77431 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.15 avg-zval = 648.40 usrkeys = 77431 avg-key = 16.00 avg-zkey = 1.70 avg-val = 652.15 avg-zval = 648.40 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 77431 bytes = 203264 } seq expand size = 542017 multi value expand size = 0 cnt WriteAppend = 6137 UnZipSize{ index = 0.0012 GB value = 0.0500 GB dict = 0.00 MB all = 0.0518 GB } __ZipSize{ index = 0.0001 GB value = 0.0502 GB dict = 0.00 MB all = 0.0505 GB } UnZip/Zip{ index = 9.4061 value = 0.9950 dict = 0.00 all = 1.0251 } Zip/UnZip{ index = 0.1063 value = 1.0050 dict = 0.00 all = 0.9755 } ---------------------------- total value len = 13.082659 GB avg = 0.262 KB (by entry num) total key len = 1.288652 GB avg = 0.026 KB total ukey len = 0.888830 GB avg = 0.018 KB total ukey num = 0.049977712 Billion total entry num = 0.049977712 Billion write speed all = 39.012214440 MB/sec (with seq num) write speed all = 37.926862521 MB/sec (without seq num) 2024/11/19-15:24:45.350691 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005177.sst): second pass time = 1.00's, 129.958'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 342.809'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2561.128'MB/sec (index lex order gen) rebuild zvType time = 0.08's, 0.376'MB/sec write SST data time = 0.03's, 1175.983'MB/sec dict compress time = 0.00's, 1995.925'MB/sec z-dict build time = 0.00's, sample length = 1.355'MB, throughput = 18966.187'MB/sec zip my value time = 1.00's, unzip length = 0.129'GB zip my value throughput = 129.958'MB/sec zip pipeline throughput = 129.958'MB/sec entries = 127197 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.01 avg-zval = 309.86 usrkeys = 127197 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.01 avg-zval = 309.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0015 GB value = 0.1294 GB dict = 1.36 MB all = 0.1319 GB } __ZipSize{ index = 0.0002 GB value = 0.0394 GB dict = 1.36 MB all = 0.0410 GB } UnZip/Zip{ index = 7.0980 value = 3.2822 dict = 1.00 all = 3.2184 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 13.212020 GB avg = 0.264 KB (by entry num) total key len = 1.291196 GB avg = 0.026 KB total ukey len = 0.890356 GB avg = 0.018 KB total ukey num = 0.050104909 Billion total entry num = 0.050104909 Billion write speed all = 39.369988101 MB/sec (with seq num) write speed all = 38.281882000 MB/sec (without seq num) 2024/11/19-15:24:45.350878 139850978166528 [compaction_job.cc:1948] [default] [JOB 208] Generated table #5178: 77431 keys, 50543257 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:45.350935 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001085350903, "cf_name": "default", "job": 208, "event": "table_file_creation", "file_number": 5178, "file_size": 50543257, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138271341, "largest_seqno": 139657750, "table_properties": {"data_size": 50206320, "index_size": 131712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1858344, "raw_average_key_size": 24, "raw_value_size": 49954449, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 77431, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001085, "oldest_key_time": 0, "file_creation_time": 1732001085, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5178, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:45.351767 139851179493120 [compaction_job.cc:1948] [default] [JOB 204] Generated table #5177: 127197 keys, 40984834 bytes, temperature: kUnknown 2024/11/19-15:24:45.351808 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085351788, "cf_name": "default", "job": 204, "event": "table_file_creation", "file_number": 5177, "file_size": 40984834, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 39412768, "index_size": 215040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2543940, "raw_average_key_size": 20, "raw_value_size": 129360873, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 127197, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001084, "oldest_key_time": 0, "file_creation_time": 1732001084, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5177, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:45.363165 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005179.sst) /storage/mytopling/..//tmp/Topling-82038-GNUbia 2024/11/19-15:24:45.372877 139851179493120 [compaction_job.cc:703] job-00204: subcompact[0], size: 0.082685 G, files: /storage/mytopling/.rocksdb [005176,005177] 2024/11/19-15:24:45.374754 139851179493120 (Original Log Time 2024/11/19-15:24:42.691329) EVENT_LOG_v1 {"time_micros": 1732001082691305, "job": 204, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4623, 4624, 4625, 4783], "score": 2.02504, "input_data_size": 262047388, "oldest_snapshot_seqno": 141581927} 2024/11/19-15:24:45.374764 139851179493120 (Original Log Time 2024/11/19-15:24:45.374263) [compaction_job.cc:2022] [default] [JOB 204] Compacted 4@2 files to L3 => 82684592 bytes 2024/11/19-15:24:45.374767 139851179493120 (Original Log Time 2024/11/19-15:24:45.374597) [compaction_job.cc:1224] [default] compacted to: files[3 132 54 55 0 0 0] max score 6.65 (165 files need compaction), MB/sec: 97.7 rd, 30.8 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(249.9, 0.0 +0.0 blob) out(78.9 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 256626, records dropped: 0 output_compression: Snappy 2024/11/19-15:24:45.374772 139851179493120 (Original Log Time 2024/11/19-15:24:45.374622) EVENT_LOG_v1 {"time_micros": 1732001085374609, "cf": "default", "job": 204, "event": "compaction_finished", "compaction_time_micros": 2681561, "compaction_time_cpu_micros": 1120872, "output_level": 3, "num_output_files": 2, "total_output_size": 82684592, "num_input_records": 256626, "num_output_records": 256626, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 132, 54, 55, 0, 0, 0]} 2024/11/19-15:24:45.376213 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-02 2024/11/19-15:24:45.376315 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00207/att-02/compact.done) = No such file or directory: 7.205 sec on http probe failed: 3, retry 2024/11/19-15:24:45.380139 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004621.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.380164 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085380161, "job": 204, "event": "table_file_deletion", "file_number": 4621} 2024/11/19-15:24:45.381046 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004891.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.381065 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085381062, "job": 204, "event": "table_file_deletion", "file_number": 4891} 2024/11/19-15:24:45.381937 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004892.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.381951 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085381950, "job": 204, "event": "table_file_deletion", "file_number": 4892} 2024/11/19-15:24:45.382803 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004894.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.382815 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085382813, "job": 204, "event": "table_file_deletion", "file_number": 4894} 2024/11/19-15:24:45.383668 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004895.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.383682 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085383680, "job": 204, "event": "table_file_deletion", "file_number": 4895} 2024/11/19-15:24:45.384780 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004897.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.384797 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085384795, "job": 204, "event": "table_file_deletion", "file_number": 4897} 2024/11/19-15:24:45.385711 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004898.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.385724 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085385722, "job": 204, "event": "table_file_deletion", "file_number": 4898} 2024/11/19-15:24:45.386621 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004900.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.386639 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085386637, "job": 204, "event": "table_file_deletion", "file_number": 4900} 2024/11/19-15:24:45.387702 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004901.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.387720 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085387718, "job": 204, "event": "table_file_deletion", "file_number": 4901} 2024/11/19-15:24:45.388639 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004902.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.388654 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085388652, "job": 204, "event": "table_file_deletion", "file_number": 4902} 2024/11/19-15:24:45.391273 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-01 2024/11/19-15:24:45.391364 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00211/att-01/compact.done) = No such file or directory: 11.231 sec on http probe failed: 5, giveup 2024/11/19-15:24:45.391382 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00211/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.817 sec, real 11.495 (submit 259.051 ms wait 11.231 s), input 127.817 MiB, send shutdown 2024/11/19-15:24:45.400063 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-01 2024/11/19-15:24:45.400272 139851169007360 [dcompact_etcd.cc:1773] job-00211/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:45.457498 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-02 2024/11/19-15:24:45.457577 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00209/att-02/compact.done) = No such file or directory: 7.845 sec on http probe failed: 4, retry 2024/11/19-15:24:45.461400 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004904.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.461430 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085461426, "job": 204, "event": "table_file_deletion", "file_number": 4904} 2024/11/19-15:24:45.462320 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004906.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.462336 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085462334, "job": 204, "event": "table_file_deletion", "file_number": 4906} 2024/11/19-15:24:45.463207 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004907.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.463221 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085463220, "job": 204, "event": "table_file_deletion", "file_number": 4907} 2024/11/19-15:24:45.464066 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004909.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.464079 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085464078, "job": 204, "event": "table_file_deletion", "file_number": 4909} 2024/11/19-15:24:45.464927 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004910.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.464940 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085464938, "job": 204, "event": "table_file_deletion", "file_number": 4910} 2024/11/19-15:24:45.465772 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004911.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.465784 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085465782, "job": 204, "event": "table_file_deletion", "file_number": 4911} 2024/11/19-15:24:45.466629 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004913.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.466642 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085466640, "job": 204, "event": "table_file_deletion", "file_number": 4913} 2024/11/19-15:24:45.467640 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004914.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.467655 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085467653, "job": 204, "event": "table_file_deletion", "file_number": 4914} 2024/11/19-15:24:45.468531 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004915.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.468544 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085468543, "job": 204, "event": "table_file_deletion", "file_number": 4915} 2024/11/19-15:24:45.469390 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004917.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.469402 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085469401, "job": 204, "event": "table_file_deletion", "file_number": 4917} 2024/11/19-15:24:45.470247 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004918.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.470259 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085470258, "job": 204, "event": "table_file_deletion", "file_number": 4918} 2024/11/19-15:24:45.471097 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004919.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.471109 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085471108, "job": 204, "event": "table_file_deletion", "file_number": 4919} 2024/11/19-15:24:45.471946 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004921.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.471958 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085471957, "job": 204, "event": "table_file_deletion", "file_number": 4921} 2024/11/19-15:24:45.472796 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004922.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.472807 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085472806, "job": 204, "event": "table_file_deletion", "file_number": 4922} 2024/11/19-15:24:45.473710 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004923.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.473723 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085473722, "job": 204, "event": "table_file_deletion", "file_number": 4923} 2024/11/19-15:24:45.474591 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004924.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.474604 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085474602, "job": 204, "event": "table_file_deletion", "file_number": 4924} 2024/11/19-15:24:45.475455 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004925.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.475467 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085475466, "job": 204, "event": "table_file_deletion", "file_number": 4925} 2024/11/19-15:24:45.476314 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004926.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.476327 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085476326, "job": 204, "event": "table_file_deletion", "file_number": 4926} 2024/11/19-15:24:45.477178 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004927.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.477191 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085477189, "job": 204, "event": "table_file_deletion", "file_number": 4927} 2024/11/19-15:24:45.478024 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004928.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.478036 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085478035, "job": 204, "event": "table_file_deletion", "file_number": 4928} 2024/11/19-15:24:45.478862 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004929.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.478874 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085478873, "job": 204, "event": "table_file_deletion", "file_number": 4929} 2024/11/19-15:24:45.479706 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004930.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.479718 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085479717, "job": 204, "event": "table_file_deletion", "file_number": 4930} 2024/11/19-15:24:45.480886 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004931.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.480902 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085480901, "job": 204, "event": "table_file_deletion", "file_number": 4931} 2024/11/19-15:24:45.481736 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004932.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.481749 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085481747, "job": 204, "event": "table_file_deletion", "file_number": 4932} 2024/11/19-15:24:45.482592 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004933.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.482605 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085482603, "job": 204, "event": "table_file_deletion", "file_number": 4933} 2024/11/19-15:24:45.483458 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004934.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.483470 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085483469, "job": 204, "event": "table_file_deletion", "file_number": 4934} 2024/11/19-15:24:45.484197 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004935.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.484209 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085484207, "job": 204, "event": "table_file_deletion", "file_number": 4935} 2024/11/19-15:24:45.521152 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005179.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.043 GB 2024/11/19-15:24:45.521242 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005179.sst): index pass time = 0.00's,64749.111'MB/sec index type = UintIndex_AllOne usrkeys = 576000 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0046 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:24:45.550386 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004943.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.550419 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085550416, "job": 204, "event": "table_file_deletion", "file_number": 4943} 2024/11/19-15:24:45.569193 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005179.sst): first pass time = 0.21's, 206.917'MB/sec 2024/11/19-15:24:45.569236 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005179.sst): old prealloc_size = 73819750, real_size = 33052104 2024/11/19-15:24:45.569258 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005179.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 576000 prefix = 4 raw-key = 0.0046 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0311 GB zip-val = 0.0329 GB avg-val = 54.01 avg-zval = 57.13 2024/11/19-15:24:45.622166 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004961.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.622207 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085622202, "job": 204, "event": "table_file_deletion", "file_number": 4961} 2024/11/19-15:24:45.646827 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005179.sst): second pass time = 0.00's,178778781.609'MB/sec, value only(73.0% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.08's, 446.490'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.463'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 3.957'MB/sec write SST data time = 0.04's, 841.326'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.031'GB zip my value throughput = 178778781.609'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 576000 avg-key = 20.00 avg-zkey = 0.00 avg-val = 54.01 avg-zval = 57.13 usrkeys = 576000 avg-key = 12.00 avg-zkey = 0.00 avg-val = 61.01 avg-zval = 57.13 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 576000 bytes = 1728016 } seq expand size = 4032000 multi value expand size = 0 cnt WriteAppend = 4025 UnZipSize{ index = 0.0069 GB value = 0.0311 GB dict = 0.00 MB all = 0.0426 GB } __ZipSize{ index = 0.0000 GB value = 0.0329 GB dict = 0.00 MB all = 0.0346 GB } UnZip/Zip{ index =36000.0000 value = 0.9453 dict = 0.00 all = 1.2307 } Zip/UnZip{ index = 0.0000 value = 1.0579 dict = 0.00 all = 0.8126 } ---------------------------- total value len = 13.247159 GB avg = 0.261 KB (by entry num) total key len = 1.302716 GB avg = 0.026 KB total ukey len = 0.897268 GB avg = 0.018 KB total ukey num = 0.050680909 Billion total entry num = 0.050680909 Billion write speed all = 39.464923569 MB/sec (with seq num) write speed all = 38.365192782 MB/sec (without seq num) 2024/11/19-15:24:45.649272 139850978166528 [compaction_job.cc:1948] [default] [JOB 208] Generated table #5179: 576000 keys, 34637141 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:45.649319 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001085649297, "cf_name": "default", "job": 208, "event": "table_file_creation", "file_number": 5179, "file_size": 34637141, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130209950, "largest_seqno": 139660751, "table_properties": {"data_size": 32907728, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 11520000, "raw_average_key_size": 20, "raw_value_size": 31107508, "raw_average_value_size": 54, "num_data_blocks": 1, "num_entries": 576000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001085, "oldest_key_time": 0, "file_creation_time": 1732001085, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5179, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:45.658657 139850978166528 [compaction_job.cc:703] job-00208: subcompact[0], size: 0.085180 G, files: /storage/mytopling/.rocksdb [005178,005179] 2024/11/19-15:24:45.659661 139850978166528 (Original Log Time 2024/11/19-15:24:45.180913) EVENT_LOG_v1 {"time_micros": 1732001085180892, "job": 208, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5044, 5047, 5049, 5016], "score": 7.7591, "input_data_size": 97407686, "oldest_snapshot_seqno": 142380812} 2024/11/19-15:24:45.659670 139850978166528 (Original Log Time 2024/11/19-15:24:45.659189) [compaction_job.cc:2022] [default] [JOB 208] Compacted 4@1 files to L2 => 85180398 bytes 2024/11/19-15:24:45.659674 139850978166528 (Original Log Time 2024/11/19-15:24:45.659518) [compaction_job.cc:1224] [default] compacted to: files[3 128 56 55 0 0 0] max score 6.65 (167 files need compaction), MB/sec: 203.9 rd, 178.3 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(92.9, 0.0 +0.0 blob) out(81.2 +0.0 blob), read-write-amplify(1.9) write-amplify(0.9) OK, records in: 653431, records dropped: 0 output_compression: Snappy 2024/11/19-15:24:45.659689 139850978166528 (Original Log Time 2024/11/19-15:24:45.659541) EVENT_LOG_v1 {"time_micros": 1732001085659529, "cf": "default", "job": 208, "event": "compaction_finished", "compaction_time_micros": 477756, "compaction_time_cpu_micros": 474051, "output_level": 2, "num_output_files": 2, "total_output_size": 85180398, "num_input_records": 653431, "num_output_records": 653431, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 128, 56, 55, 0, 0, 0]} 2024/11/19-15:24:45.659766 139850978166528 [compaction_job.cc:2437] [default] [JOB 215] Compacting 4@1 files to L2, score 6.65, subcompactions 1 : 1 2024/11/19-15:24:45.659775 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 214 Base level 1, inputs: [5102(31MB) 5104(31MB) 5106(31MB) 5108(31MB)] 2024/11/19-15:24:45.698574 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004966.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.698599 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085698596, "job": 204, "event": "table_file_deletion", "file_number": 4966} 2024/11/19-15:24:45.703303 139851169007360 [dcompact_etcd.cc:1163] job-00211/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.303 sec, input = 127.817 MiB 2024/11/19-15:24:45.706595 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004972.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.706625 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085706621, "job": 204, "event": "table_file_deletion", "file_number": 4972} 2024/11/19-15:24:45.715347 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004973.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.715372 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085715369, "job": 204, "event": "table_file_deletion", "file_number": 4973} 2024/11/19-15:24:45.724015 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004974.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.724038 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085724036, "job": 204, "event": "table_file_deletion", "file_number": 4974} 2024/11/19-15:24:45.731567 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004975.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.731588 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085731586, "job": 204, "event": "table_file_deletion", "file_number": 4975} 2024/11/19-15:24:45.799521 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004977.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.799556 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085799553, "job": 204, "event": "table_file_deletion", "file_number": 4977} 2024/11/19-15:24:45.869430 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004985.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.869471 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085869467, "job": 204, "event": "table_file_deletion", "file_number": 4985} 2024/11/19-15:24:45.959025 139850978166528 [dcompact_etcd.cc:1163] job-00215/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.299 sec, input = 127.820 MiB 2024/11/19-15:24:45.966724 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005004.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:45.966763 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001085966758, "job": 204, "event": "table_file_deletion", "file_number": 5004} 2024/11/19-15:24:45.966989 139851179493120 [compaction_job.cc:2437] [default] [JOB 216] Compacting 4@1 files to L2, score 6.40, subcompactions 1 : 1 2024/11/19-15:24:45.967000 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 214 Base level 1, inputs: [5113(31MB) 5118(31MB) 5123(31MB) 5128(31MB)] 2024/11/19-15:24:46.271724 139851179493120 [dcompact_etcd.cc:1163] job-00216/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.304 sec, input = 127.822 MiB 2024/11/19-15:24:46.347453 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-00 2024/11/19-15:24:46.347528 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00214/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:24:46.869940 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-01 2024/11/19-15:24:46.870037 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00212/att-01/compact.done) = No such file or directory: 9.223 sec on http probe failed: 4, retry 2024/11/19-15:24:47.306096 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-02 2024/11/19-15:24:47.306191 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00209/att-02/compact.done) = No such file or directory: 9.693 sec on http probe failed: 5, giveup 2024/11/19-15:24:47.306209 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00209/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 91.966 sec, real 9.995 (submit 292.880 ms wait 9.693 s), input 91.966 MiB, send shutdown 2024/11/19-15:24:47.313934 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00209/att-02 2024/11/19-15:24:47.314163 139851158521600 [dcompact_etcd.cc:1773] job-00209/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:47.314313 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00209) = success 2024/11/19-15:24:47.314319 139851158521600 [dcompact_etcd.cc:1773] job-00209/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:47.314336 139851158521600 (Original Log Time 2024/11/19-15:24:17.174908) EVENT_LOG_v1 {"time_micros": 1732001057174902, "job": 209, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5024, 5028, 5038, 5046], "score": 7.57767, "input_data_size": 96694615, "oldest_snapshot_seqno": 142466576} 2024/11/19-15:24:47.314344 139851158521600 [compaction_job.cc:2437] [default] [JOB 209] Compacting 4@1 files to L2, score 7.58, subcompactions 1 : 1 2024/11/19-15:24:47.314352 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 207 Base level 1, inputs: [5024(4437KB) 5028(24MB) 5038(31MB) 5046(31MB)] 2024/11/19-15:24:47.314555 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005180.sst) /storage/mytopling/..//tmp/Topling-82038-J01AFK 2024/11/19-15:24:47.356515 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005180.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0016 GB, waited 0.000 sec, Key+Value = 0.004 GB 2024/11/19-15:24:47.358005 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005180.sst): first pass time = 0.04's, 96.468'MB/sec 2024/11/19-15:24:47.365693 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005180.sst): index pass time = 0.01's, 228.995'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 174600 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0021 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 2024/11/19-15:24:47.365758 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005180.sst): old prealloc_size = 73819750, real_size = 339680 2024/11/19-15:24:47.365859 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005180.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZeroLengthBlobStore usrkeys = 174600 prefix = 4 raw-key = 0.0021 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 0.00 avg-zval = 0.00 2024/11/19-15:24:47.365976 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005180.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.009 sec, Key+Value = 0.004 GB 2024/11/19-15:24:47.381181 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-02 2024/11/19-15:24:47.381269 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00207/att-02/compact.done) = No such file or directory: 9.214 sec on http probe failed: 4, retry 2024/11/19-15:24:47.394251 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005180.sst): second pass time = 0.00's, 0.000'MB/sec, value only( 0.0% of KV) wait indexing time = 0.01's, remap KeyValue time = 0.03's, 28.866'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3073.495'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 307.976'MB/sec write SST data time = 0.03's, 29.108'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 0.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 174600 avg-key = 24.00 avg-zkey = 1.69 avg-val = 0.00 avg-zval = 0.00 usrkeys = 174600 avg-key = 16.00 avg-zkey = 1.69 avg-val = 7.00 avg-zval = 0.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 174600 bytes = 523808 } seq expand size = 1222200 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0028 GB value = 0.0000 GB dict = 0.00 MB all = 0.0042 GB } __ZipSize{ index = 0.0003 GB value = 0.0000 GB dict = 0.00 MB all = 0.0008 GB } UnZip/Zip{ index = 9.4460 value = 0.0000 dict = 0.00 all = 5.0987 } Zip/UnZip{ index = 0.1059 value = inf dict = 0.00 all = 0.1961 } ---------------------------- total value len = 13.248382 GB avg = 0.261 KB (by entry num) total key len = 1.306906 GB avg = 0.026 KB total ukey len = 0.900062 GB avg = 0.018 KB total ukey num = 0.050855509 Billion total entry num = 0.050855509 Billion write speed all = 39.293367437 MB/sec (with seq num) write speed all = 38.195053619 MB/sec (without seq num) 2024/11/19-15:24:47.394980 139851158521600 [compaction_job.cc:1948] [default] [JOB 209] Generated table #5180: 174600 keys, 821862 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:47.395041 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001087395009, "cf_name": "default", "job": 209, "event": "table_file_creation", "file_number": 5180, "file_size": 821862, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130202142, "largest_seqno": 139714807, "table_properties": {"data_size": 80, "index_size": 295744, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4190400, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 174600, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZeroLengthBlobStore", "creation_time": 1732001087, "oldest_key_time": 0, "file_creation_time": 1732001087, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5180, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:47.396023 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005181.sst) /storage/mytopling/..//tmp/Topling-82038-nsxf2h 2024/11/19-15:24:47.546432 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005181.sst: sumWaitingMem = 0.005 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0052 GB, waited 0.000 sec, Key+Value = 0.025 GB 2024/11/19-15:24:47.576483 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005181.sst): index pass time = 0.03's, 230.278'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 576000 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0069 GB zip-key = 0.0010 GB avg-key = 12.00 avg-zkey = 1.68 2024/11/19-15:24:47.581616 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005181.sst): first pass time = 0.19's, 135.948'MB/sec 2024/11/19-15:24:47.581663 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005181.sst): old prealloc_size = 73819750, real_size = 13083904 2024/11/19-15:24:47.581926 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005181.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 576000 prefix = 4 raw-key = 0.0069 GB zip-key = 0.0010 GB avg-key = 12.00 avg-zkey = 1.68 raw-val = 0.0114 GB zip-val = 0.0120 GB avg-val = 19.80 avg-zval = 20.78 2024/11/19-15:24:47.582247 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005181.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0014 GB, waited 0.036 sec, Key+Value = 0.025 GB 2024/11/19-15:24:47.652345 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005181.sst): second pass time = 0.00's,52800000.000'MB/sec, value only(45.2% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 207.645'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3711.875'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 7.019'MB/sec write SST data time = 0.05's, 294.109'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.011'GB zip my value throughput = 52800000.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 576000 avg-key = 24.00 avg-zkey = 1.68 avg-val = 19.80 avg-zval = 20.78 usrkeys = 576000 avg-key = 16.00 avg-zkey = 1.68 avg-val = 26.80 avg-zval = 20.78 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 576000 bytes = 1728016 } seq expand size = 4032000 multi value expand size = 0 cnt WriteAppend = 1404 UnZipSize{ index = 0.0092 GB value = 0.0114 GB dict = 0.00 MB all = 0.0252 GB } __ZipSize{ index = 0.0010 GB value = 0.0120 GB dict = 0.00 MB all = 0.0147 GB } UnZip/Zip{ index = 9.5068 value = 0.9528 dict = 0.00 all = 1.7193 } Zip/UnZip{ index = 0.1052 value = 1.0496 dict = 0.00 all = 0.5816 } ---------------------------- total value len = 13.263818 GB avg = 0.258 KB (by entry num) total key len = 1.320730 GB avg = 0.026 KB total ukey len = 0.909278 GB avg = 0.018 KB total ukey num = 0.051431509 Billion total entry num = 0.051431509 Billion write speed all = 39.344945085 MB/sec (with seq num) write speed all = 38.234964946 MB/sec (without seq num) 2024/11/19-15:24:47.653312 139851158521600 [compaction_job.cc:1948] [default] [JOB 209] Generated table #5181: 576000 keys, 14673693 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:47.653361 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001087653337, "cf_name": "default", "job": 209, "event": "table_file_creation", "file_number": 5181, "file_size": 14673693, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130215952, "largest_seqno": 139673753, "table_properties": {"data_size": 11970288, "index_size": 969408, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13824000, "raw_average_key_size": 24, "raw_value_size": 11404800, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 576000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001087, "oldest_key_time": 0, "file_creation_time": 1732001087, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5181, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:47.658044 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005182.sst) /storage/mytopling/..//tmp/Topling-82038-0JltLV 2024/11/19-15:24:47.806139 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005182.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:24:47.856489 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005182.sst): index pass time = 0.05's, 147.150'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462690 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:24:47.918894 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005182.sst): first pass time = 0.26's, 256.925'MB/sec 2024/11/19-15:24:47.918957 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005182.sst): old prealloc_size = 73819750, real_size = 55526752 2024/11/19-15:24:47.919189 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005182.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462690 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0545 GB avg-val = 116.84 avg-zval = 117.87 2024/11/19-15:24:47.919435 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005182.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.113 sec, Key+Value = 0.067 GB 2024/11/19-15:24:48.016446 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005182.sst): second pass time = 0.00's,162829036.145'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 580.406'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3817.904'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 2.547'MB/sec write SST data time = 0.05's, 1091.452'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 162829036.145'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462690 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.84 avg-zval = 117.87 usrkeys = 462690 avg-key = 20.00 avg-zkey = 1.89 avg-val = 123.84 avg-zval = 117.87 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 462690 bytes = 1156736 } seq expand size = 3238830 multi value expand size = 0 cnt WriteAppend = 6611 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0670 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0566 GB } UnZip/Zip{ index = 10.5710 value = 0.9913 dict = 0.00 all = 1.1846 } Zip/UnZip{ index = 0.0946 value = 1.0088 dict = 0.00 all = 0.8442 } ---------------------------- total value len = 13.321117 GB avg = 0.257 KB (by entry num) total key len = 1.333685 GB avg = 0.026 KB total ukey len = 0.918532 GB avg = 0.018 KB total ukey num = 0.051894199 Billion total entry num = 0.051894199 Billion write speed all = 39.495674755 MB/sec (with seq num) write speed all = 38.376807960 MB/sec (without seq num) 2024/11/19-15:24:48.019990 139851158521600 [compaction_job.cc:1948] [default] [JOB 209] Generated table #5182: 462690 keys, 56573221 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:48.020041 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001088020018, "cf_name": "default", "job": 209, "event": "table_file_creation", "file_number": 5182, "file_size": 56573221, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 130221954, "largest_seqno": 131073071, "table_properties": {"data_size": 54535504, "index_size": 875392, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12955320, "raw_average_key_size": 28, "raw_value_size": 54059240, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462690, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001087, "oldest_key_time": 0, "file_creation_time": 1732001087, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5182, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:48.034588 139851158521600 [compaction_job.cc:703] job-00209: subcompact[0], size: 0.072069 G, files: /storage/mytopling/.rocksdb [005180,005181,005182] 2024/11/19-15:24:48.036182 139851158521600 (Original Log Time 2024/11/19-15:24:47.314378) EVENT_LOG_v1 {"time_micros": 1732001087314363, "job": 209, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5024, 5028, 5038, 5046], "score": 7.57767, "input_data_size": 96694615, "oldest_snapshot_seqno": 142466576} 2024/11/19-15:24:48.036193 139851158521600 (Original Log Time 2024/11/19-15:24:48.035706) [compaction_job.cc:2022] [default] [JOB 209] Compacted 4@1 files to L2 => 72068776 bytes 2024/11/19-15:24:48.036196 139851158521600 (Original Log Time 2024/11/19-15:24:48.036066) [compaction_job.cc:1224] [default] compacted to: files[3 124 59 55 0 0 0] max score 6.15 (162 files need compaction), MB/sec: 134.3 rd, 100.1 wr, level 2, files in(4, 0) out(3 +0 blob) MB in(92.2, 0.0 +0.0 blob) out(68.7 +0.0 blob), read-write-amplify(1.7) write-amplify(0.7) OK, records in: 1213290, records dropped: 0 output_compression: Snappy 2024/11/19-15:24:48.036213 139851158521600 (Original Log Time 2024/11/19-15:24:48.036092) EVENT_LOG_v1 {"time_micros": 1732001088036079, "cf": "default", "job": 209, "event": "compaction_finished", "compaction_time_micros": 720223, "compaction_time_cpu_micros": 708339, "output_level": 2, "num_output_files": 3, "total_output_size": 72068776, "num_input_records": 1213290, "num_output_records": 1213290, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 124, 59, 55, 0, 0, 0]} 2024/11/19-15:24:48.036287 139851158521600 [compaction_job.cc:2437] [default] [JOB 217] Compacting 3@1 files to L2, score 6.15, subcompactions 1 : 1 2024/11/19-15:24:48.036296 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 215 Base level 1, inputs: [5133(31MB) 5137(31MB) 5142(28MB)] 2024/11/19-15:24:48.311467 139851158521600 [dcompact_etcd.cc:1163] job-00217/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.275 sec, input = 92.146 MiB 2024/11/19-15:24:48.353511 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-00 2024/11/19-15:24:48.353575 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00214/att-00/compact.done) = No such file or directory: 5.201 sec on http probe failed: 2, retry 2024/11/19-15:24:48.878611 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-01 2024/11/19-15:24:48.878691 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00212/att-01/compact.done) = No such file or directory: 11.229 sec on http probe failed: 5, giveup 2024/11/19-15:24:48.878710 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00212/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.823 sec, real 11.515 (submit 277.803 ms wait 11.229 s), input 127.823 MiB, send shutdown 2024/11/19-15:24:48.887206 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-01 2024/11/19-15:24:48.887427 139851137550080 [dcompact_etcd.cc:1773] job-00212/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:48.907451 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-02 2024/11/19-15:24:48.907529 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00211/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:24:49.169242 139851137550080 [dcompact_etcd.cc:1163] job-00212/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.282 sec, input = 127.823 MiB 2024/11/19-15:24:49.172185 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-00 2024/11/19-15:24:49.172252 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00215/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:24:49.390325 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-02 2024/11/19-15:24:49.390398 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00207/att-02/compact.done) = No such file or directory: 11.219 sec on http probe failed: 5, giveup 2024/11/19-15:24:49.390417 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00207/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.238 sec, real 11.512 (submit 283.567 ms wait 11.219 s), input 127.238 MiB, send shutdown 2024/11/19-15:24:49.398059 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00207/att-02 2024/11/19-15:24:49.398389 139851127064320 [dcompact_etcd.cc:1773] job-00207/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:49.398621 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00207) = success 2024/11/19-15:24:49.398627 139851127064320 [dcompact_etcd.cc:1773] job-00207/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:49.398643 139851127064320 (Original Log Time 2024/11/19-15:24:14.808759) EVENT_LOG_v1 {"time_micros": 1732001054808750, "job": 207, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5034, 5037, 5040, 5042], "score": 8.00875, "input_data_size": 134027712, "oldest_snapshot_seqno": 142307062} 2024/11/19-15:24:49.398650 139851127064320 [compaction_job.cc:2437] [default] [JOB 207] Compacting 4@1 files to L2, score 8.01, subcompactions 1 : 1 2024/11/19-15:24:49.398658 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 205 Base level 1, inputs: [5034(31MB) 5037(31MB) 5040(31MB) 5042(31MB)] 2024/11/19-15:24:49.398896 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005183.sst) /storage/mytopling/..//tmp/Topling-82038-vgW9St 2024/11/19-15:24:49.473608 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005183.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:24:49.478092 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-00 2024/11/19-15:24:49.478211 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00216/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:24:49.478977 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005183.sst): index pass time = 0.01's, 225.416'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 100290 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:24:49.543021 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005183.sst): first pass time = 0.14's, 465.677'MB/sec 2024/11/19-15:24:49.543072 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005183.sst): old prealloc_size = 73819750, real_size = 65223728 2024/11/19-15:24:49.543145 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005183.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 100290 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0650 GB avg-val = 645.15 avg-zval = 648.40 2024/11/19-15:24:49.543214 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005183.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.070 sec, Key+Value = 0.067 GB 2024/11/19-15:24:49.612052 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005183.sst): second pass time = 0.00's,278888879.310'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 945.445'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2450.617'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.554'MB/sec write SST data time = 0.02's, 2759.103'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 278888879.310'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 100290 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.15 avg-zval = 648.40 usrkeys = 100290 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.15 avg-zval = 648.40 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7943 UnZipSize{ index = 0.0016 GB value = 0.0647 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0002 GB value = 0.0650 GB dict = 0.00 MB all = 0.0652 GB } UnZip/Zip{ index = 9.4328 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 13.385819 GB avg = 0.257 KB (by entry num) total key len = 1.336092 GB avg = 0.026 KB total ukey len = 0.920136 GB avg = 0.018 KB total ukey num = 0.051994489 Billion total entry num = 0.051994489 Billion write speed all = 39.506649252 MB/sec (with seq num) write speed all = 38.390420236 MB/sec (without seq num) 2024/11/19-15:24:49.616006 139851127064320 [compaction_job.cc:1948] [default] [JOB 207] Generated table #5183: 100290 keys, 65200589 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:49.616058 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001089616034, "cf_name": "default", "job": 207, "event": "table_file_creation", "file_number": 5183, "file_size": 65200589, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 65028384, "index_size": 170112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2406960, "raw_average_key_size": 24, "raw_value_size": 64702220, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 100290, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001089, "oldest_key_time": 0, "file_creation_time": 1732001089, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5183, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:49.631851 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005184.sst) /storage/mytopling/..//tmp/Topling-82038-pngdKZ 2024/11/19-15:24:49.691644 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005184.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:24:49.696962 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005184.sst): index pass time = 0.01's, 225.206'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 99154 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:24:49.761775 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005184.sst): first pass time = 0.13's, 510.440'MB/sec 2024/11/19-15:24:49.761826 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005184.sst): old prealloc_size = 73819750, real_size = 64445248 2024/11/19-15:24:49.761897 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005184.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 99154 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0639 GB zip-val = 0.0643 GB avg-val = 644.75 avg-zval = 648.00 2024/11/19-15:24:49.761974 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005184.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.070 sec, Key+Value = 0.066 GB 2024/11/19-15:24:49.831613 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005184.sst): second pass time = 0.00's,387452187.879'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 923.746'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2462.079'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.557'MB/sec write SST data time = 0.03's, 2559.828'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.064'GB zip my value throughput = 387452187.879'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 99154 avg-key = 24.00 avg-zkey = 1.70 avg-val = 644.75 avg-zval = 648.00 usrkeys = 99154 avg-key = 16.00 avg-zkey = 1.70 avg-val = 644.75 avg-zval = 648.00 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7848 UnZipSize{ index = 0.0016 GB value = 0.0639 GB dict = 0.00 MB all = 0.0663 GB } __ZipSize{ index = 0.0002 GB value = 0.0643 GB dict = 0.00 MB all = 0.0644 GB } UnZip/Zip{ index = 9.4325 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9715 } ---------------------------- total value len = 13.449748 GB avg = 0.258 KB (by entry num) total key len = 1.338472 GB avg = 0.026 KB total ukey len = 0.921723 GB avg = 0.018 KB total ukey num = 0.052093643 Billion total entry num = 0.052093643 Billion write speed all = 39.661226996 MB/sec (with seq num) write speed all = 38.543527775 MB/sec (without seq num) 2024/11/19-15:24:49.852147 139851127064320 [compaction_job.cc:1948] [default] [JOB 207] Generated table #5184: 99154 keys, 64422349 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:49.852203 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001089852180, "cf_name": "default", "job": 207, "event": "table_file_creation", "file_number": 5184, "file_size": 64422349, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64252080, "index_size": 168192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2379696, "raw_average_key_size": 24, "raw_value_size": 63929611, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 99154, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001089, "oldest_key_time": 0, "file_creation_time": 1732001089, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5184, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:49.868287 139851127064320 [compaction_job.cc:703] job-00207: subcompact[0], size: 0.129623 G, files: /storage/mytopling/.rocksdb [005183,005184] 2024/11/19-15:24:49.873277 139851127064320 (Original Log Time 2024/11/19-15:24:49.398686) EVENT_LOG_v1 {"time_micros": 1732001089398670, "job": 207, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5034, 5037, 5040, 5042], "score": 8.00875, "input_data_size": 134027712, "oldest_snapshot_seqno": 142307062} 2024/11/19-15:24:49.873288 139851127064320 (Original Log Time 2024/11/19-15:24:49.872735) [compaction_job.cc:2022] [default] [JOB 207] Compacted 4@1 files to L2 => 129622938 bytes 2024/11/19-15:24:49.873292 139851127064320 (Original Log Time 2024/11/19-15:24:49.873144) [compaction_job.cc:1224] [default] compacted to: files[3 120 61 55 0 0 0] max score 5.97 (161 files need compaction), MB/sec: 285.4 rd, 276.0 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.8, 0.0 +0.0 blob) out(123.6 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 199444, records dropped: 0 output_compression: Snappy 2024/11/19-15:24:49.873310 139851127064320 (Original Log Time 2024/11/19-15:24:49.873173) EVENT_LOG_v1 {"time_micros": 1732001089873159, "cf": "default", "job": 207, "event": "compaction_finished", "compaction_time_micros": 469617, "compaction_time_cpu_micros": 452066, "output_level": 2, "num_output_files": 2, "total_output_size": 129622938, "num_input_records": 199444, "num_output_records": 199444, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 120, 61, 55, 0, 0, 0]} 2024/11/19-15:24:49.892576 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004607.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:49.892606 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001089892602, "job": 207, "event": "table_file_deletion", "file_number": 4607} 2024/11/19-15:24:49.898391 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004608.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:49.898416 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001089898413, "job": 207, "event": "table_file_deletion", "file_number": 4608} 2024/11/19-15:24:49.904730 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004609.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:49.904752 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001089904749, "job": 207, "event": "table_file_deletion", "file_number": 4609} 2024/11/19-15:24:49.910816 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004610.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:49.910837 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001089910835, "job": 207, "event": "table_file_deletion", "file_number": 4610} 2024/11/19-15:24:49.916097 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004611.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:49.916118 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001089916115, "job": 207, "event": "table_file_deletion", "file_number": 4611} 2024/11/19-15:24:49.921691 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004612.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:49.921711 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001089921709, "job": 207, "event": "table_file_deletion", "file_number": 4612} 2024/11/19-15:24:49.927334 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004613.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:49.927355 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001089927353, "job": 207, "event": "table_file_deletion", "file_number": 4613} 2024/11/19-15:24:49.932250 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004620.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:49.932270 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001089932268, "job": 207, "event": "table_file_deletion", "file_number": 4620} 2024/11/19-15:24:49.938427 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004967.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:49.938447 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001089938445, "job": 207, "event": "table_file_deletion", "file_number": 4967} 2024/11/19-15:24:49.945245 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004978.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:49.945266 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001089945264, "job": 207, "event": "table_file_deletion", "file_number": 4978} 2024/11/19-15:24:49.952280 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004979.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:49.952300 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001089952298, "job": 207, "event": "table_file_deletion", "file_number": 4979} 2024/11/19-15:24:49.952454 139851127064320 [compaction_job.cc:2437] [default] [JOB 218] Compacting 4@1 files to L2, score 5.97, subcompactions 1 : 1 2024/11/19-15:24:49.952462 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 216 Base level 1, inputs: [5146(31MB) 5148(31MB) 5150(3231KB) 5017(31MB)] 2024/11/19-15:24:50.261076 139851127064320 [dcompact_etcd.cc:1163] job-00218/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.308 sec, input = 98.280 MiB 2024/11/19-15:24:50.358953 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-00 2024/11/19-15:24:50.359025 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00214/att-00/compact.done) = No such file or directory: 7.207 sec on http probe failed: 3, retry 2024/11/19-15:24:50.624673 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-00 2024/11/19-15:24:50.624771 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00217/att-00/compact.done) = No such file or directory: 2.304 sec on http probe failed: 1, retry 2024/11/19-15:24:50.912269 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-02 2024/11/19-15:24:50.912343 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00211/att-02/compact.done) = No such file or directory: 5.204 sec on http probe failed: 2, retry 2024/11/19-15:24:51.182055 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-00 2024/11/19-15:24:51.182148 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00215/att-00/compact.done) = No such file or directory: 5.213 sec on http probe failed: 2, retry 2024/11/19-15:24:51.487652 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-00 2024/11/19-15:24:51.487755 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00216/att-00/compact.done) = No such file or directory: 5.207 sec on http probe failed: 2, retry 2024/11/19-15:24:52.367351 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-00 2024/11/19-15:24:52.367451 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00214/att-00/compact.done) = No such file or directory: 9.213 sec on http probe failed: 4, retry 2024/11/19-15:24:52.373930 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-02 2024/11/19-15:24:52.373999 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00212/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:24:52.476432 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-00 2024/11/19-15:24:52.476509 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00217/att-00/compact.done) = No such file or directory: 4.156 sec on http probe failed: 2, retry 2024/11/19-15:24:52.728307 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-00 2024/11/19-15:24:52.728408 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00218/att-00/compact.done) = No such file or directory: 2.457 sec on http probe failed: 1, retry 2024/11/19-15:24:52.917977 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-02 2024/11/19-15:24:52.918070 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00211/att-02/compact.done) = No such file or directory: 7.209 sec on http probe failed: 3, retry 2024/11/19-15:24:53.190118 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-00 2024/11/19-15:24:53.190199 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00215/att-00/compact.done) = No such file or directory: 7.223 sec on http probe failed: 3, retry 2024/11/19-15:24:53.498022 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-00 2024/11/19-15:24:53.498131 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00216/att-00/compact.done) = No such file or directory: 7.216 sec on http probe failed: 3, retry 2024/11/19-15:24:54.328738 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-00 2024/11/19-15:24:54.328816 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00217/att-00/compact.done) = No such file or directory: 6.008 sec on http probe failed: 3, retry 2024/11/19-15:24:54.375363 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-00 2024/11/19-15:24:54.375450 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00214/att-00/compact.done) = No such file or directory: 11.221 sec on http probe failed: 5, giveup 2024/11/19-15:24:54.375468 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00214/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.822 sec, real 11.534 (submit 305.220 ms wait 11.221 s), input 127.822 MiB, send shutdown 2024/11/19-15:24:54.379372 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-02 2024/11/19-15:24:54.379435 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00212/att-02/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:24:54.381314 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-00 2024/11/19-15:24:54.391667 139851148035840 [dcompact_etcd.cc:1773] job-00214/att-00: CleanFiles 0.010 sec : 0.000 2024/11/19-15:24:54.660870 139851148035840 [dcompact_etcd.cc:1163] job-00214/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.269 sec, input = 127.822 MiB 2024/11/19-15:24:54.702513 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-00 2024/11/19-15:24:54.702582 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00218/att-00/compact.done) = No such file or directory: 4.433 sec on http probe failed: 2, retry 2024/11/19-15:24:54.926725 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-02 2024/11/19-15:24:54.926824 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00211/att-02/compact.done) = No such file or directory: 9.215 sec on http probe failed: 4, retry 2024/11/19-15:24:55.199617 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-00 2024/11/19-15:24:55.199768 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00215/att-00/compact.done) = No such file or directory: 9.231 sec on http probe failed: 4, retry 2024/11/19-15:24:55.504881 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-00 2024/11/19-15:24:55.504972 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00216/att-00/compact.done) = No such file or directory: 9.227 sec on http probe failed: 4, retry 2024/11/19-15:24:56.180717 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-00 2024/11/19-15:24:56.180815 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00217/att-00/compact.done) = No such file or directory: 7.860 sec on http probe failed: 4, retry 2024/11/19-15:24:56.388469 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-02 2024/11/19-15:24:56.388539 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00212/att-02/compact.done) = No such file or directory: 7.210 sec on http probe failed: 3, retry 2024/11/19-15:24:56.677456 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-00 2024/11/19-15:24:56.677555 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00218/att-00/compact.done) = No such file or directory: 6.407 sec on http probe failed: 3, retry 2024/11/19-15:24:56.937100 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-02 2024/11/19-15:24:56.937234 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00211/att-02/compact.done) = No such file or directory: 11.224 sec on http probe failed: 5, giveup 2024/11/19-15:24:56.937273 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00211/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.817 sec, real 11.537 (submit 302.961 ms wait 11.224 s), input 127.817 MiB, send shutdown 2024/11/19-15:24:56.944727 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00211/att-02 2024/11/19-15:24:56.944997 139851169007360 [dcompact_etcd.cc:1773] job-00211/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:56.946876 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00211) = success 2024/11/19-15:24:56.946894 139851169007360 [dcompact_etcd.cc:1773] job-00211/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:24:56.946933 139851169007360 (Original Log Time 2024/11/19-15:24:22.336718) EVENT_LOG_v1 {"time_micros": 1732001062336710, "job": 211, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5054, 5063, 5067, 5072], "score": 7.39688, "input_data_size": 133300172, "oldest_snapshot_seqno": -1} 2024/11/19-15:24:56.946944 139851169007360 [compaction_job.cc:2437] [default] [JOB 211] Compacting 4@1 files to L2, score 7.40, subcompactions 1 : 1 2024/11/19-15:24:56.946962 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 209 Base level 1, inputs: [5054(31MB) 5063(31MB) 5067(31MB) 5072(31MB)] 2024/11/19-15:24:56.947220 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005185.sst) /storage/mytopling/..//tmp/Topling-82038-uUD89V 2024/11/19-15:24:57.090604 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005185.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:24:57.124536 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005185.sst): index pass time = 0.03's, 218.826'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463516 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:24:57.202424 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005185.sst): first pass time = 0.26's, 262.983'MB/sec 2024/11/19-15:24:57.202477 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005185.sst): old prealloc_size = 73819750, real_size = 55619520 2024/11/19-15:24:57.202725 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005185.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463516 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.78 avg-zval = 117.85 2024/11/19-15:24:57.202780 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005185.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.112 sec, Key+Value = 0.067 GB 2024/11/19-15:24:57.208871 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-00 2024/11/19-15:24:57.208955 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00215/att-00/compact.done) = No such file or directory: 11.241 sec on http probe failed: 5, giveup 2024/11/19-15:24:57.208973 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00215/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.549 (submit 298.795 ms wait 11.241 s), input 127.820 MiB, send shutdown 2024/11/19-15:24:57.217245 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-00 2024/11/19-15:24:57.217470 139850978166528 [dcompact_etcd.cc:1773] job-00215/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:57.297832 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005185.sst): second pass time = 0.00's,136348846.348'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 582.223'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3558.817'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 2.562'MB/sec write SST data time = 0.05's, 1113.113'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 136348846.348'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463516 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 117.85 usrkeys = 463516 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6615 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5714 value = 0.9909 dict = 0.00 all = 1.2090 } Zip/UnZip{ index = 0.0946 value = 1.0092 dict = 0.00 all = 0.8271 } ---------------------------- total value len = 13.503879 GB avg = 0.257 KB (by entry num) total key len = 1.351450 GB avg = 0.026 KB total ukey len = 0.930993 GB avg = 0.018 KB total ukey num = 0.052557159 Billion total entry num = 0.052557159 Billion write speed all = 39.059087228 MB/sec (with seq num) write speed all = 37.953579776 MB/sec (without seq num) 2024/11/19-15:24:57.301635 139851169007360 [compaction_job.cc:1948] [default] [JOB 211] Generated table #5185: 463516 keys, 55508952 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:57.301687 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001097301664, "cf_name": "default", "job": 211, "event": "table_file_creation", "file_number": 5185, "file_size": 55508952, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54626544, "index_size": 876928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12978448, "raw_average_key_size": 28, "raw_value_size": 54130492, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463516, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001096, "oldest_key_time": 0, "file_creation_time": 1732001096, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5185, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:57.316441 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005186.sst) /storage/mytopling/..//tmp/Topling-82038-byBhgC 2024/11/19-15:24:57.458732 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005186.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:24:57.481089 139850978166528 [dcompact_etcd.cc:1163] job-00215/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.264 sec, input = 127.820 MiB 2024/11/19-15:24:57.492847 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005186.sst): index pass time = 0.03's, 216.992'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462122 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:24:57.510278 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-00 2024/11/19-15:24:57.510360 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00216/att-00/compact.done) = No such file or directory: 11.233 sec on http probe failed: 5, giveup 2024/11/19-15:24:57.510384 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00216/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.822 sec, real 11.543 (submit 304.250 ms wait 11.233 s), input 127.822 MiB, send shutdown 2024/11/19-15:24:57.517802 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-00 2024/11/19-15:24:57.518070 139851179493120 [dcompact_etcd.cc:1773] job-00216/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:57.561780 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005186.sst): first pass time = 0.25's, 272.773'MB/sec 2024/11/19-15:24:57.561831 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005186.sst): old prealloc_size = 73819750, real_size = 55454992 2024/11/19-15:24:57.562070 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005186.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462122 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.80 avg-zval = 117.86 2024/11/19-15:24:57.562135 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005186.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.103 sec, Key+Value = 0.067 GB 2024/11/19-15:24:57.656781 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005186.sst): second pass time = 0.00's,97963078.040'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 582.974'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3704.895'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.587'MB/sec write SST data time = 0.05's, 1105.808'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 97963078.040'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462122 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.80 avg-zval = 117.86 usrkeys = 462122 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.80 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6596 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0553 GB } UnZip/Zip{ index = 10.5689 value = 0.9911 dict = 0.00 all = 1.2091 } Zip/UnZip{ index = 0.0946 value = 1.0090 dict = 0.00 all = 0.8271 } ---------------------------- total value len = 13.557857 GB avg = 0.256 KB (by entry num) total key len = 1.364390 GB avg = 0.026 KB total ukey len = 0.940236 GB avg = 0.018 KB total ukey num = 0.053019281 Billion total entry num = 0.053019281 Billion write speed all = 39.198037482 MB/sec (with seq num) write speed all = 38.083861126 MB/sec (without seq num) 2024/11/19-15:24:57.660385 139851169007360 [compaction_job.cc:1948] [default] [JOB 211] Generated table #5186: 462122 keys, 55344728 bytes (need compaction), temperature: kUnknown 2024/11/19-15:24:57.660436 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001097660413, "cf_name": "default", "job": 211, "event": "table_file_creation", "file_number": 5186, "file_size": 55344728, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54464768, "index_size": 874496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12939416, "raw_average_key_size": 28, "raw_value_size": 53977656, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462122, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001097, "oldest_key_time": 0, "file_creation_time": 1732001097, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5186, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:24:57.674236 139851169007360 [compaction_job.cc:703] job-00211: subcompact[0], size: 0.110854 G, files: /storage/mytopling/.rocksdb [005185,005186] 2024/11/19-15:24:57.675618 139851169007360 (Original Log Time 2024/11/19-15:24:56.946999) EVENT_LOG_v1 {"time_micros": 1732001096946978, "job": 211, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5054, 5063, 5067, 5072], "score": 7.39688, "input_data_size": 133300172, "oldest_snapshot_seqno": -1} 2024/11/19-15:24:57.675630 139851169007360 (Original Log Time 2024/11/19-15:24:57.675150) [compaction_job.cc:2022] [default] [JOB 211] Compacted 4@1 files to L2 => 110853680 bytes 2024/11/19-15:24:57.675633 139851169007360 (Original Log Time 2024/11/19-15:24:57.675485) [compaction_job.cc:1224] [default] compacted to: files[3 116 63 55 0 0 0] max score 5.78 (159 files need compaction), MB/sec: 183.3 rd, 152.4 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925638, records dropped: 0 output_compression: Snappy 2024/11/19-15:24:57.675652 139851169007360 (Original Log Time 2024/11/19-15:24:57.675509) EVENT_LOG_v1 {"time_micros": 1732001097675496, "cf": "default", "job": 211, "event": "compaction_finished", "compaction_time_micros": 727251, "compaction_time_cpu_micros": 723994, "output_level": 2, "num_output_files": 2, "total_output_size": 110853680, "num_input_records": 925638, "num_output_records": 925638, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 116, 63, 55, 0, 0, 0]} 2024/11/19-15:24:57.675838 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005015.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:57.675853 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001097675851, "job": 211, "event": "table_file_deletion", "file_number": 5015} 2024/11/19-15:24:57.676069 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005018.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:57.676080 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001097676078, "job": 211, "event": "table_file_deletion", "file_number": 5018} 2024/11/19-15:24:57.680175 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005019.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:57.680196 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001097680193, "job": 211, "event": "table_file_deletion", "file_number": 5019} 2024/11/19-15:24:57.684292 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005021.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:24:57.684312 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001097684310, "job": 211, "event": "table_file_deletion", "file_number": 5021} 2024/11/19-15:24:57.684449 139851169007360 [compaction_job.cc:2437] [default] [JOB 219] Compacting 4@1 files to L2, score 5.78, subcompactions 1 : 1 2024/11/19-15:24:57.684458 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 217 Base level 1, inputs: [5020(31MB) 5022(31MB) 5025(31MB) 5027(31MB)] 2024/11/19-15:24:57.780887 139852349699840 [db_impl_write.cc:2181] [default] New memtable created with log file: #5187. Immutable memtables: 0. 2024/11/19-15:24:57.781531 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 220] Syncing log #5172 2024/11/19-15:24:57.781638 139850957195008 (Original Log Time 2024/11/19-15:24:57.781272) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:24:57.781651 139850957195008 [flush_job.cc:879] [default] [JOB 220] Flushing memtable with next log file: 5187 2024/11/19-15:24:57.781679 139850957195008 EVENT_LOG_v1 {"time_micros": 1732001097781671, "job": 220, "event": "flush_started", "num_memtables": 1, "num_entries": 1583072, "num_deletes": 0, "total_data_size": 476415489, "memory_usage": 545783176, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:24:57.781686 139850957195008 [flush_job.cc:910] [default] [JOB 220] Level-0 flush table #5188: started 2024/11/19-15:24:57.782425 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000028.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/005188.sst in ConvertToSST, mem_size = 520.499 M, time = 726.979 us 2024/11/19-15:24:57.820805 139851179493120 [dcompact_etcd.cc:1163] job-00216/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.303 sec, input = 127.822 MiB 2024/11/19-15:24:57.858585 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/005188.sst): fsize = 520.501 M, time(ms): open: 0.042, seek: 0.018, finish: 0.051, meta: 0.015, Flush: 0.015, sync: 75.988, close: 0.008, all: 76.137 2024/11/19-15:24:57.859703 139850957195008 [flush_job.cc:1083] [default] [JOB 220] Flush lasted 78065 microseconds, and 57018 cpu microseconds. 2024/11/19-15:24:57.865666 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-01 2024/11/19-15:24:57.865735 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00214/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:24:57.882605 139850957195008 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:24:57.882672 139850957195008 (Original Log Time 2024/11/19-15:24:57.858631) [flush_job.cc:1029] [default] [JOB 220] Level-0 flush table #5188: 545784602 bytes OK (needs compaction) 2024/11/19-15:24:57.882677 139850957195008 (Original Log Time 2024/11/19-15:24:57.859718) [memtable_list.cc:560] [default] Level-0 commit table #5188 started 2024/11/19-15:24:57.882680 139850957195008 (Original Log Time 2024/11/19-15:24:57.882576) [memtable_list.cc:759] [default] Level-0 commit table #5188: memtable #1 done 2024/11/19-15:24:57.882683 139850957195008 (Original Log Time 2024/11/19-15:24:57.882600) EVENT_LOG_v1 {"time_micros": 1732001097882595, "job": 220, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [4, 116, 63, 55, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:24:57.882686 139850957195008 (Original Log Time 2024/11/19-15:24:57.882647) [db_impl_compaction_flush.cc:310] [default] Level summary: files[4 116 63 55 0 0 0] max score 5.53 (156 files need compaction) 2024/11/19-15:24:57.958903 139851169007360 [dcompact_etcd.cc:1163] job-00219/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.274 sec, input = 126.878 MiB 2024/11/19-15:24:58.029169 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-00 2024/11/19-15:24:58.029260 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00217/att-00/compact.done) = No such file or directory: 9.712 sec on http probe failed: 5, giveup 2024/11/19-15:24:58.029278 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00217/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 92.146 sec, real 9.993 (submit 274.750 ms wait 9.712 s), input 92.146 MiB, send shutdown 2024/11/19-15:24:58.037555 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-00 2024/11/19-15:24:58.037838 139851158521600 [dcompact_etcd.cc:1773] job-00217/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:24:58.313501 139851158521600 [dcompact_etcd.cc:1163] job-00217/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.276 sec, input = 92.146 MiB 2024/11/19-15:24:58.397476 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-02 2024/11/19-15:24:58.397558 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00212/att-02/compact.done) = No such file or directory: 9.219 sec on http probe failed: 4, retry 2024/11/19-15:24:58.652103 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-00 2024/11/19-15:24:58.652199 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00218/att-00/compact.done) = No such file or directory: 8.382 sec on http probe failed: 4, retry 2024/11/19-15:24:59.870662 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-01 2024/11/19-15:24:59.870757 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00214/att-01/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:25:00.405893 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-02 2024/11/19-15:25:00.405977 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00212/att-02/compact.done) = No such file or directory: 11.228 sec on http probe failed: 5, giveup 2024/11/19-15:25:00.405997 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00212/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.823 sec, real 11.518 (submit 281.737 ms wait 11.228 s), input 127.823 MiB, send shutdown 2024/11/19-15:25:00.410489 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00212/att-02 2024/11/19-15:25:00.410725 139851137550080 [dcompact_etcd.cc:1773] job-00212/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:00.412602 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00212) = success 2024/11/19-15:25:00.412611 139851137550080 [dcompact_etcd.cc:1773] job-00212/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:25:00.412629 139851137550080 (Original Log Time 2024/11/19-15:24:25.811100) EVENT_LOG_v1 {"time_micros": 1732001065811095, "job": 212, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5076, 5080, 5084, 5088], "score": 7.14724, "input_data_size": 133300172, "oldest_snapshot_seqno": -1} 2024/11/19-15:25:00.412636 139851137550080 [compaction_job.cc:2437] [default] [JOB 212] Compacting 4@1 files to L2, score 7.15, subcompactions 1 : 1 2024/11/19-15:25:00.412644 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 210 Base level 1, inputs: [5076(31MB) 5080(31MB) 5084(31MB) 5088(31MB)] 2024/11/19-15:25:00.412889 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005189.sst) /storage/mytopling/..//tmp/Topling-82038-aVf8Y6 2024/11/19-15:25:00.555649 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005189.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:00.589664 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005189.sst): index pass time = 0.03's, 218.202'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463406 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:25:00.623117 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-00 2024/11/19-15:25:00.623189 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00218/att-00/compact.done) = No such file or directory: 10.357 sec on http probe failed: 5, giveup 2024/11/19-15:25:00.623206 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00218/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 98.280 sec, real 10.670 (submit 308.191 ms wait 10.357 s), input 98.280 MiB, send shutdown 2024/11/19-15:25:00.626178 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-01 2024/11/19-15:25:00.626243 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00217/att-01/compact.done) = No such file or directory: 2.304 sec on http probe failed: 1, retry 2024/11/19-15:25:00.630716 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-00 2024/11/19-15:25:00.630935 139851127064320 [dcompact_etcd.cc:1773] job-00218/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:00.657053 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005189.sst): first pass time = 0.24's, 274.865'MB/sec 2024/11/19-15:25:00.657098 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005189.sst): old prealloc_size = 73819750, real_size = 55610112 2024/11/19-15:25:00.657324 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005189.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463406 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.82 avg-zval = 117.86 2024/11/19-15:25:00.657394 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005189.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.102 sec, Key+Value = 0.067 GB 2024/11/19-15:25:00.686563 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-01 2024/11/19-15:25:00.686662 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00215/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:25:00.751931 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005189.sst): second pass time = 0.00's,234344658.009'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 585.393'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3965.112'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.619'MB/sec write SST data time = 0.05's, 1102.133'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 234344658.009'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463406 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.82 avg-zval = 117.86 usrkeys = 463406 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.82 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6616 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5743 value = 0.9911 dict = 0.00 all = 1.2092 } Zip/UnZip{ index = 0.0946 value = 1.0089 dict = 0.00 all = 0.8270 } ---------------------------- total value len = 13.611990 GB avg = 0.255 KB (by entry num) total key len = 1.377365 GB avg = 0.026 KB total ukey len = 0.949504 GB avg = 0.018 KB total ukey num = 0.053482687 Billion total entry num = 0.053482687 Billion write speed all = 39.056774498 MB/sec (with seq num) write speed all = 37.941924023 MB/sec (without seq num) 2024/11/19-15:25:00.755657 139851137550080 [compaction_job.cc:1948] [default] [JOB 212] Generated table #5189: 463406 keys, 55499544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:00.755711 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001100755687, "cf_name": "default", "job": 212, "event": "table_file_creation", "file_number": 5189, "file_size": 55499544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54617584, "index_size": 876480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12975368, "raw_average_key_size": 28, "raw_value_size": 54133616, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463406, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001100, "oldest_key_time": 0, "file_creation_time": 1732001100, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5189, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:00.769663 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005190.sst) /storage/mytopling/..//tmp/Topling-82038-gNNDzn 2024/11/19-15:25:00.910914 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005190.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:00.920440 139851127064320 [dcompact_etcd.cc:1163] job-00218/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 98.280 MiB 2024/11/19-15:25:00.960374 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005190.sst): index pass time = 0.05's, 149.690'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462276 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.92 2024/11/19-15:25:01.025522 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-01 2024/11/19-15:25:01.025636 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00216/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:25:01.025890 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005190.sst): first pass time = 0.26's, 261.207'MB/sec 2024/11/19-15:25:01.025948 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005190.sst): old prealloc_size = 73819750, real_size = 55480624 2024/11/19-15:25:01.026178 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005190.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462276 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.77 avg-zval = 117.85 2024/11/19-15:25:01.026233 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005190.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.115 sec, Key+Value = 0.067 GB 2024/11/19-15:25:01.119393 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005190.sst): second pass time = 0.00's,249905648.148'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 592.582'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3925.340'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.619'MB/sec write SST data time = 0.05's, 1127.837'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 249905648.148'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462276 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.77 avg-zval = 117.85 usrkeys = 462276 avg-key = 20.00 avg-zkey = 1.92 avg-val = 116.77 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6597 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0554 GB } UnZip/Zip{ index = 10.4349 value = 0.9908 dict = 0.00 all = 1.2086 } Zip/UnZip{ index = 0.0958 value = 1.0092 dict = 0.00 all = 0.8274 } ---------------------------- total value len = 13.665970 GB avg = 0.253 KB (by entry num) total key len = 1.390309 GB avg = 0.026 KB total ukey len = 0.958749 GB avg = 0.018 KB total ukey num = 0.053944963 Billion total entry num = 0.053944963 Billion write speed all = 39.193624547 MB/sec (with seq num) write speed all = 38.070213556 MB/sec (without seq num) 2024/11/19-15:25:01.122961 139851137550080 [compaction_job.cc:1948] [default] [JOB 212] Generated table #5190: 462276 keys, 55370392 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:01.123011 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001101122989, "cf_name": "default", "job": 212, "event": "table_file_creation", "file_number": 5190, "file_size": 55370392, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54478880, "index_size": 886016, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12943728, "raw_average_key_size": 28, "raw_value_size": 53979620, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462276, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001100, "oldest_key_time": 0, "file_creation_time": 1732001100, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5190, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:01.136913 139851137550080 [compaction_job.cc:703] job-00212: subcompact[0], size: 0.110870 G, files: /storage/mytopling/.rocksdb [005189,005190] 2024/11/19-15:25:01.138175 139851137550080 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:25:01.138293 139851137550080 (Original Log Time 2024/11/19-15:25:00.412671) EVENT_LOG_v1 {"time_micros": 1732001100412655, "job": 212, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5076, 5080, 5084, 5088], "score": 7.14724, "input_data_size": 133300172, "oldest_snapshot_seqno": -1} 2024/11/19-15:25:01.138299 139851137550080 (Original Log Time 2024/11/19-15:25:01.137801) [compaction_job.cc:2022] [default] [JOB 212] Compacted 4@1 files to L2 => 110869936 bytes 2024/11/19-15:25:01.138303 139851137550080 (Original Log Time 2024/11/19-15:25:01.138146) [compaction_job.cc:1224] [default] compacted to: files[4 112 65 55 0 0 0] max score 5.53 (158 files need compaction), MB/sec: 184.0 rd, 153.1 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925682, records dropped: 0 output_compression: Snappy 2024/11/19-15:25:01.138319 139851137550080 (Original Log Time 2024/11/19-15:25:01.138169) EVENT_LOG_v1 {"time_micros": 1732001101138158, "cf": "default", "job": 212, "event": "compaction_finished", "compaction_time_micros": 724262, "compaction_time_cpu_micros": 721067, "output_level": 2, "num_output_files": 2, "total_output_size": 110869936, "num_input_records": 925682, "num_output_records": 925682, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 112, 65, 55, 0, 0, 0]} 2024/11/19-15:25:01.140224 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-00 2024/11/19-15:25:01.140302 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00219/att-00/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:25:01.143643 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005023.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:01.143668 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001101143665, "job": 212, "event": "table_file_deletion", "file_number": 5023} 2024/11/19-15:25:01.147933 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005026.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:01.147955 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001101147952, "job": 212, "event": "table_file_deletion", "file_number": 5026} 2024/11/19-15:25:01.152503 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005030.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:01.152523 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001101152521, "job": 212, "event": "table_file_deletion", "file_number": 5030} 2024/11/19-15:25:01.156853 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005032.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:01.156872 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001101156870, "job": 212, "event": "table_file_deletion", "file_number": 5032} 2024/11/19-15:25:01.157008 139851137550080 [compaction_job.cc:2437] [default] [JOB 221] Compacting 4@1 files to L2, score 5.53, subcompactions 1 : 1 2024/11/19-15:25:01.157017 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 219 Base level 1, inputs: [5029(31MB) 5031(31MB) 5033(31MB) 5035(31MB)] 2024/11/19-15:25:01.464374 139851137550080 [dcompact_etcd.cc:1163] job-00221/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.307 sec, input = 126.908 MiB 2024/11/19-15:25:01.880188 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-01 2024/11/19-15:25:01.880292 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00214/att-01/compact.done) = No such file or directory: 7.210 sec on http probe failed: 3, retry 2024/11/19-15:25:02.477763 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-01 2024/11/19-15:25:02.477841 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00217/att-01/compact.done) = No such file or directory: 4.156 sec on http probe failed: 2, retry 2024/11/19-15:25:02.695360 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-01 2024/11/19-15:25:02.695469 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00215/att-01/compact.done) = No such file or directory: 5.206 sec on http probe failed: 2, retry 2024/11/19-15:25:03.033961 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-01 2024/11/19-15:25:03.034042 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00216/att-01/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:25:03.150131 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-00 2024/11/19-15:25:03.150211 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00219/att-00/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:25:03.387696 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-01 2024/11/19-15:25:03.387808 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00218/att-01/compact.done) = No such file or directory: 2.457 sec on http probe failed: 1, retry 2024/11/19-15:25:03.888972 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-01 2024/11/19-15:25:03.889091 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00214/att-01/compact.done) = No such file or directory: 9.220 sec on http probe failed: 4, retry 2024/11/19-15:25:04.331475 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-01 2024/11/19-15:25:04.331580 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00217/att-01/compact.done) = No such file or directory: 6.007 sec on http probe failed: 3, retry 2024/11/19-15:25:04.646445 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-00 2024/11/19-15:25:04.646542 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00221/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:25:04.704180 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-01 2024/11/19-15:25:04.704241 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00215/att-01/compact.done) = No such file or directory: 7.214 sec on http probe failed: 3, retry 2024/11/19-15:25:05.042811 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-01 2024/11/19-15:25:05.042899 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00216/att-01/compact.done) = No such file or directory: 7.213 sec on http probe failed: 3, retry 2024/11/19-15:25:05.156088 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-00 2024/11/19-15:25:05.156164 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00219/att-00/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:25:05.358853 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-01 2024/11/19-15:25:05.358926 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00218/att-01/compact.done) = No such file or directory: 4.433 sec on http probe failed: 2, retry 2024/11/19-15:25:05.894137 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-01 2024/11/19-15:25:05.894300 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00214/att-01/compact.done) = No such file or directory: 11.228 sec on http probe failed: 5, giveup 2024/11/19-15:25:05.894328 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00214/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.822 sec, real 11.503 (submit 269.117 ms wait 11.228 s), input 127.822 MiB, send shutdown 2024/11/19-15:25:05.901135 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-01 2024/11/19-15:25:05.901395 139851148035840 [dcompact_etcd.cc:1773] job-00214/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:06.095956 139852362282752 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:25:06.183737 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-01 2024/11/19-15:25:06.183832 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00217/att-01/compact.done) = No such file or directory: 7.861 sec on http probe failed: 4, retry 2024/11/19-15:25:06.208986 139851148035840 [dcompact_etcd.cc:1163] job-00214/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.307 sec, input = 127.822 MiB 2024/11/19-15:25:06.655706 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-00 2024/11/19-15:25:06.655783 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00221/att-00/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:25:06.708783 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-01 2024/11/19-15:25:06.708859 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00215/att-01/compact.done) = No such file or directory: 9.223 sec on http probe failed: 4, retry 2024/11/19-15:25:07.050693 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-01 2024/11/19-15:25:07.050773 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00216/att-01/compact.done) = No such file or directory: 9.222 sec on http probe failed: 4, retry 2024/11/19-15:25:07.165648 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-00 2024/11/19-15:25:07.165726 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00219/att-00/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:25:07.330478 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-01 2024/11/19-15:25:07.330566 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00218/att-01/compact.done) = No such file or directory: 6.404 sec on http probe failed: 3, retry 2024/11/19-15:25:08.034895 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-01 2024/11/19-15:25:08.034999 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00217/att-01/compact.done) = No such file or directory: 9.713 sec on http probe failed: 5, giveup 2024/11/19-15:25:08.035017 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00217/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 92.146 sec, real 9.997 (submit 275.579 ms wait 9.713 s), input 92.146 MiB, send shutdown 2024/11/19-15:25:08.043461 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-01 2024/11/19-15:25:08.043684 139851158521600 [dcompact_etcd.cc:1773] job-00217/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:08.334687 139851158521600 [dcompact_etcd.cc:1163] job-00217/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.291 sec, input = 92.146 MiB 2024/11/19-15:25:08.663697 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-00 2024/11/19-15:25:08.663772 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00221/att-00/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:25:08.714886 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-01 2024/11/19-15:25:08.714956 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00215/att-01/compact.done) = No such file or directory: 11.228 sec on http probe failed: 5, giveup 2024/11/19-15:25:08.714975 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00215/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.497 (submit 263.547 ms wait 11.228 s), input 127.820 MiB, send shutdown 2024/11/19-15:25:08.721907 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-01 2024/11/19-15:25:08.722130 139850978166528 [dcompact_etcd.cc:1773] job-00215/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:09.034435 139850978166528 [dcompact_etcd.cc:1163] job-00215/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.312 sec, input = 127.820 MiB 2024/11/19-15:25:09.060752 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-01 2024/11/19-15:25:09.060813 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00216/att-01/compact.done) = No such file or directory: 11.230 sec on http probe failed: 5, giveup 2024/11/19-15:25:09.060830 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00216/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.822 sec, real 11.543 (submit 302.632 ms wait 11.230 s), input 127.822 MiB, send shutdown 2024/11/19-15:25:09.069516 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-01 2024/11/19-15:25:09.069712 139851179493120 [dcompact_etcd.cc:1773] job-00216/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:09.174077 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-00 2024/11/19-15:25:09.174182 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00219/att-00/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:25:09.174202 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00219/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.878 sec, real 11.489 (submit 274.013 ms wait 11.207 s), input 126.878 MiB, send shutdown 2024/11/19-15:25:09.182948 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-00 2024/11/19-15:25:09.183170 139851169007360 [dcompact_etcd.cc:1773] job-00219/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:09.303950 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-01 2024/11/19-15:25:09.304047 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00218/att-01/compact.done) = No such file or directory: 8.376 sec on http probe failed: 4, retry 2024/11/19-15:25:09.361951 139851179493120 [dcompact_etcd.cc:1163] job-00216/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.292 sec, input = 127.822 MiB 2024/11/19-15:25:09.412346 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-02 2024/11/19-15:25:09.412415 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00214/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:25:09.487646 139851169007360 [dcompact_etcd.cc:1163] job-00219/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.304 sec, input = 126.878 MiB 2024/11/19-15:25:10.648165 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-02 2024/11/19-15:25:10.648261 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00217/att-02/compact.done) = No such file or directory: 2.304 sec on http probe failed: 1, retry 2024/11/19-15:25:10.672454 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-00 2024/11/19-15:25:10.672518 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00221/att-00/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:25:11.274650 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-01 2024/11/19-15:25:11.274731 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00218/att-01/compact.done) = No such file or directory: 10.349 sec on http probe failed: 5, giveup 2024/11/19-15:25:11.274750 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00218/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 98.280 sec, real 10.644 (submit 289.429 ms wait 10.349 s), input 98.280 MiB, send shutdown 2024/11/19-15:25:11.283394 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-01 2024/11/19-15:25:11.283604 139851127064320 [dcompact_etcd.cc:1773] job-00218/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:11.421344 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-02 2024/11/19-15:25:11.421432 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00214/att-02/compact.done) = No such file or directory: 5.203 sec on http probe failed: 2, retry 2024/11/19-15:25:11.556229 139851127064320 [dcompact_etcd.cc:1163] job-00218/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.273 sec, input = 98.280 MiB 2024/11/19-15:25:12.240583 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-02 2024/11/19-15:25:12.240671 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00215/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:25:12.500646 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-02 2024/11/19-15:25:12.500747 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00217/att-02/compact.done) = No such file or directory: 4.157 sec on http probe failed: 2, retry 2024/11/19-15:25:12.565596 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-02 2024/11/19-15:25:12.565707 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00216/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:25:12.669533 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-01 2024/11/19-15:25:12.669637 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00219/att-01/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:25:12.676936 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-00 2024/11/19-15:25:12.677008 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00221/att-00/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:25:12.677025 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00221/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.908 sec, real 11.520 (submit 306.944 ms wait 11.208 s), input 126.908 MiB, send shutdown 2024/11/19-15:25:12.685812 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-00 2024/11/19-15:25:12.686031 139851137550080 [dcompact_etcd.cc:1773] job-00221/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:12.982955 139851137550080 [dcompact_etcd.cc:1163] job-00221/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.297 sec, input = 126.908 MiB 2024/11/19-15:25:13.429659 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-02 2024/11/19-15:25:13.429761 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00214/att-02/compact.done) = No such file or directory: 7.213 sec on http probe failed: 3, retry 2024/11/19-15:25:13.572213 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #5191. Immutable memtables: 0. 2024/11/19-15:25:13.572575 139852362282752 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:25:13.572823 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 222] Syncing log #5187 2024/11/19-15:25:13.572908 139850967680768 (Original Log Time 2024/11/19-15:25:13.572627) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:25:13.572916 139850967680768 [flush_job.cc:879] [default] [JOB 222] Flushing memtable with next log file: 5191 2024/11/19-15:25:13.572944 139850967680768 EVENT_LOG_v1 {"time_micros": 1732001113572937, "job": 222, "event": "flush_started", "num_memtables": 1, "num_entries": 1186327, "num_deletes": 0, "total_data_size": 472086360, "memory_usage": 536870280, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:25:13.572948 139850967680768 [flush_job.cc:910] [default] [JOB 222] Level-0 flush table #5192: started 2024/11/19-15:25:13.573210 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000029.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/005192.sst in ConvertToSST, mem_size = 511.999 M, time = 253.442 us 2024/11/19-15:25:14.021975 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-02 2024/11/19-15:25:14.022180 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00218/att-02/compact.done) = No such file or directory: 2.457 sec on http probe failed: 1, retry 2024/11/19-15:25:14.091669 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/005192.sst): fsize = 512.001 M, time(ms): open: 0.034, seek: 0.014, finish: 0.049, meta: 0.012, Flush: 0.013, sync: 518.311, close: 0.011, all: 518.444 2024/11/19-15:25:14.092748 139850967680768 [flush_job.cc:1083] [default] [JOB 222] Flush lasted 519838 microseconds, and 444577 cpu microseconds. 2024/11/19-15:25:14.113526 139850967680768 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 8202044 2024/11/19-15:25:14.113611 139850967680768 (Original Log Time 2024/11/19-15:25:14.091711) [flush_job.cc:1029] [default] [JOB 222] Level-0 flush table #5192: 536871706 bytes OK (needs compaction) 2024/11/19-15:25:14.113617 139850967680768 (Original Log Time 2024/11/19-15:25:14.092764) [memtable_list.cc:560] [default] Level-0 commit table #5192 started 2024/11/19-15:25:14.113620 139850967680768 (Original Log Time 2024/11/19-15:25:14.113495) [memtable_list.cc:759] [default] Level-0 commit table #5192: memtable #1 done 2024/11/19-15:25:14.113623 139850967680768 (Original Log Time 2024/11/19-15:25:14.113520) EVENT_LOG_v1 {"time_micros": 1732001114113514, "job": 222, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [5, 112, 65, 55, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:25:14.113626 139850967680768 (Original Log Time 2024/11/19-15:25:14.113583) [db_impl_compaction_flush.cc:310] [default] Level summary: files[5 112 65 55 0 0 0] max score 5.28 (155 files need compaction) 2024/11/19-15:25:14.249342 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-02 2024/11/19-15:25:14.249421 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00215/att-02/compact.done) = No such file or directory: 5.206 sec on http probe failed: 2, retry 2024/11/19-15:25:14.352570 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-02 2024/11/19-15:25:14.352672 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00217/att-02/compact.done) = No such file or directory: 6.009 sec on http probe failed: 3, retry 2024/11/19-15:25:14.573989 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-02 2024/11/19-15:25:14.574069 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00216/att-02/compact.done) = No such file or directory: 5.204 sec on http probe failed: 2, retry 2024/11/19-15:25:14.678661 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-01 2024/11/19-15:25:14.678726 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00219/att-01/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:25:15.435020 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-02 2024/11/19-15:25:15.435100 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00214/att-02/compact.done) = No such file or directory: 9.221 sec on http probe failed: 4, retry 2024/11/19-15:25:15.993656 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-02 2024/11/19-15:25:15.993743 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00218/att-02/compact.done) = No such file or directory: 4.432 sec on http probe failed: 2, retry 2024/11/19-15:25:16.161826 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-01 2024/11/19-15:25:16.161909 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00221/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:25:16.204993 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-02 2024/11/19-15:25:16.205054 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00217/att-02/compact.done) = No such file or directory: 7.861 sec on http probe failed: 4, retry 2024/11/19-15:25:16.258012 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-02 2024/11/19-15:25:16.258100 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00215/att-02/compact.done) = No such file or directory: 7.215 sec on http probe failed: 3, retry 2024/11/19-15:25:16.579182 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-02 2024/11/19-15:25:16.579264 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00216/att-02/compact.done) = No such file or directory: 7.212 sec on http probe failed: 3, retry 2024/11/19-15:25:16.686745 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-01 2024/11/19-15:25:16.686802 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00219/att-01/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:25:17.443453 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-02 2024/11/19-15:25:17.443549 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00214/att-02/compact.done) = No such file or directory: 11.226 sec on http probe failed: 5, giveup 2024/11/19-15:25:17.443567 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00214/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.822 sec, real 11.542 (submit 307.487 ms wait 11.226 s), input 127.822 MiB, send shutdown 2024/11/19-15:25:17.448865 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00214/att-02 2024/11/19-15:25:17.449098 139851148035840 [dcompact_etcd.cc:1773] job-00214/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:17.450953 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00214) = success 2024/11/19-15:25:17.450962 139851148035840 [dcompact_etcd.cc:1773] job-00214/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:25:17.450980 139851148035840 (Original Log Time 2024/11/19-15:24:42.840718) EVENT_LOG_v1 {"time_micros": 1732001082840712, "job": 214, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5091, 5095, 5098, 5100], "score": 6.89758, "input_data_size": 133300108, "oldest_snapshot_seqno": 144473571} 2024/11/19-15:25:17.450988 139851148035840 [compaction_job.cc:2437] [default] [JOB 214] Compacting 4@1 files to L2, score 6.90, subcompactions 1 : 1 2024/11/19-15:25:17.450995 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 212 Base level 1, inputs: [5091(31MB) 5095(31MB) 5098(31MB) 5100(31MB)] 2024/11/19-15:25:17.451207 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005193.sst) /storage/mytopling/..//tmp/Topling-82038-HfXxpQ 2024/11/19-15:25:17.594215 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005193.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:17.642823 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005193.sst): index pass time = 0.05's, 152.658'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463337 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:25:17.703492 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005193.sst): first pass time = 0.25's, 266.019'MB/sec 2024/11/19-15:25:17.703542 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005193.sst): old prealloc_size = 73819750, real_size = 55610464 2024/11/19-15:25:17.703764 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005193.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463337 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.84 avg-zval = 117.87 2024/11/19-15:25:17.703838 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005193.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.110 sec, Key+Value = 0.067 GB 2024/11/19-15:25:17.795844 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005193.sst): second pass time = 0.00's,336245701.863'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 601.409'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 4062.496'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.680'MB/sec write SST data time = 0.05's, 1136.254'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 336245701.863'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463337 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.84 avg-zval = 117.87 usrkeys = 463337 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.84 avg-zval = 117.87 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6616 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5006 value = 0.9913 dict = 0.00 all = 1.2092 } Zip/UnZip{ index = 0.0952 value = 1.0088 dict = 0.00 all = 0.8270 } ---------------------------- total value len = 13.720105 GB avg = 0.252 KB (by entry num) total key len = 1.403282 GB avg = 0.026 KB total ukey len = 0.968016 GB avg = 0.018 KB total ukey num = 0.054408300 Billion total entry num = 0.054408300 Billion write speed all = 37.730398465 MB/sec (with seq num) write speed all = 36.644479423 MB/sec (without seq num) 2024/11/19-15:25:17.799171 139851148035840 [compaction_job.cc:1948] [default] [JOB 214] Generated table #5193: 463337 keys, 55499928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:17.799232 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001117799201, "cf_name": "default", "job": 214, "event": "table_file_creation", "file_number": 5193, "file_size": 55499928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54611920, "index_size": 882496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12973436, "raw_average_key_size": 28, "raw_value_size": 54135558, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463337, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001117, "oldest_key_time": 0, "file_creation_time": 1732001117, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5193, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:17.812568 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005194.sst) /storage/mytopling/..//tmp/Topling-82038-yDPXXa 2024/11/19-15:25:17.953661 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005194.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:17.967351 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-02 2024/11/19-15:25:17.967426 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00218/att-02/compact.done) = No such file or directory: 6.403 sec on http probe failed: 3, retry 2024/11/19-15:25:17.987456 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005194.sst): index pass time = 0.03's, 219.080'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462333 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:25:18.053114 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005194.sst): first pass time = 0.24's, 278.226'MB/sec 2024/11/19-15:25:18.053166 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005194.sst): old prealloc_size = 73819750, real_size = 55473600 2024/11/19-15:25:18.053201 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-02 2024/11/19-15:25:18.053303 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00217/att-02/compact.done) = No such file or directory: 9.713 sec on http probe failed: 5, giveup 2024/11/19-15:25:18.053321 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00217/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 92.146 sec, real 10.010 (submit 290.921 ms wait 9.713 s), input 92.146 MiB, send shutdown 2024/11/19-15:25:18.053430 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005194.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462333 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.75 avg-zval = 117.84 2024/11/19-15:25:18.053504 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005194.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.100 sec, Key+Value = 0.067 GB 2024/11/19-15:25:18.058680 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00217/att-02 2024/11/19-15:25:18.058906 139851158521600 [dcompact_etcd.cc:1773] job-00217/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:18.060758 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00217) = success 2024/11/19-15:25:18.060766 139851158521600 [dcompact_etcd.cc:1773] job-00217/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:25:18.060784 139851158521600 (Original Log Time 2024/11/19-15:24:48.036308) EVENT_LOG_v1 {"time_micros": 1732001088036302, "job": 217, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5133, 5137, 5142], "score": 6.14863, "input_data_size": 96112905, "oldest_snapshot_seqno": 144814564} 2024/11/19-15:25:18.060791 139851158521600 [compaction_job.cc:2437] [default] [JOB 217] Compacting 3@1 files to L2, score 6.15, subcompactions 1 : 1 2024/11/19-15:25:18.060798 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 215 Base level 1, inputs: [5133(31MB) 5137(31MB) 5142(28MB)] 2024/11/19-15:25:18.061046 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005195.sst) /storage/mytopling/..//tmp/Topling-82038-x01ESs 2024/11/19-15:25:18.146028 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005194.sst): second pass time = 0.00's,96905583.483'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 596.298'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3326.417'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.660'MB/sec write SST data time = 0.05's, 1126.940'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 96905583.483'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462333 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.75 avg-zval = 117.84 usrkeys = 462333 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.75 avg-zval = 117.84 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6596 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0554 GB } UnZip/Zip{ index = 10.5714 value = 0.9907 dict = 0.00 all = 1.2088 } Zip/UnZip{ index = 0.0946 value = 1.0094 dict = 0.00 all = 0.8273 } ---------------------------- total value len = 13.774082 GB avg = 0.251 KB (by entry num) total key len = 1.416228 GB avg = 0.026 KB total ukey len = 0.977263 GB avg = 0.018 KB total ukey num = 0.054870633 Billion total entry num = 0.054870633 Billion write speed all = 37.864276995 MB/sec (with seq num) write speed all = 36.770086332 MB/sec (without seq num) 2024/11/19-15:25:18.149487 139851148035840 [compaction_job.cc:1948] [default] [JOB 214] Generated table #5194: 462333 keys, 55363288 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:18.149540 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001118149516, "cf_name": "default", "job": 214, "event": "table_file_creation", "file_number": 5194, "file_size": 55363288, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54483120, "index_size": 874688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12945324, "raw_average_key_size": 28, "raw_value_size": 53976410, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462333, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001117, "oldest_key_time": 0, "file_creation_time": 1732001117, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5194, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:18.162942 139851148035840 [compaction_job.cc:703] job-00214: subcompact[0], size: 0.110863 G, files: /storage/mytopling/.rocksdb [005193,005194] 2024/11/19-15:25:18.164211 139851148035840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 8455715 2024/11/19-15:25:18.164373 139851148035840 (Original Log Time 2024/11/19-15:25:17.451020) EVENT_LOG_v1 {"time_micros": 1732001117451004, "job": 214, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5091, 5095, 5098, 5100], "score": 6.89758, "input_data_size": 133300108, "oldest_snapshot_seqno": 144473571} 2024/11/19-15:25:18.164380 139851148035840 (Original Log Time 2024/11/19-15:25:18.163839) [compaction_job.cc:2022] [default] [JOB 214] Compacted 4@1 files to L2 => 110863216 bytes 2024/11/19-15:25:18.164383 139851148035840 (Original Log Time 2024/11/19-15:25:18.164181) [compaction_job.cc:1224] [default] compacted to: files[5 108 67 55 0 0 0] max score 5.28 (157 files need compaction), MB/sec: 187.2 rd, 155.7 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925670, records dropped: 0 output_compression: Snappy 2024/11/19-15:25:18.164403 139851148035840 (Original Log Time 2024/11/19-15:25:18.164205) EVENT_LOG_v1 {"time_micros": 1732001118164193, "cf": "default", "job": 214, "event": "compaction_finished", "compaction_time_micros": 711935, "compaction_time_cpu_micros": 709372, "output_level": 2, "num_output_files": 2, "total_output_size": 110863216, "num_input_records": 925670, "num_output_records": 925670, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 108, 67, 55, 0, 0, 0]} 2024/11/19-15:25:18.169467 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004623.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:18.169491 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001118169488, "job": 214, "event": "table_file_deletion", "file_number": 4623} 2024/11/19-15:25:18.171618 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-01 2024/11/19-15:25:18.171712 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00221/att-01/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:25:18.174646 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004624.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:18.174669 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001118174666, "job": 214, "event": "table_file_deletion", "file_number": 4624} 2024/11/19-15:25:18.180144 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004625.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:18.180165 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001118180163, "job": 214, "event": "table_file_deletion", "file_number": 4625} 2024/11/19-15:25:18.185287 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004783.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:18.185307 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001118185306, "job": 214, "event": "table_file_deletion", "file_number": 4783} 2024/11/19-15:25:18.188262 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005016.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:18.188280 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001118188279, "job": 214, "event": "table_file_deletion", "file_number": 5016} 2024/11/19-15:25:18.193158 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005044.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:18.193176 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001118193174, "job": 214, "event": "table_file_deletion", "file_number": 5044} 2024/11/19-15:25:18.195829 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005047.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:18.195847 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001118195845, "job": 214, "event": "table_file_deletion", "file_number": 5047} 2024/11/19-15:25:18.198829 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005049.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:18.198847 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001118198845, "job": 214, "event": "table_file_deletion", "file_number": 5049} 2024/11/19-15:25:18.199003 139851148035840 [compaction_job.cc:2437] [default] [JOB 223] Compacting 4@1 files to L2, score 5.28, subcompactions 1 : 1 2024/11/19-15:25:18.199013 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 221 Base level 1, inputs: [5036(31MB) 5039(31MB) 5041(31MB) 5043(31MB)] 2024/11/19-15:25:18.205098 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005195.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:18.239262 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005195.sst): index pass time = 0.03's, 217.227'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463564 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:25:18.266417 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-02 2024/11/19-15:25:18.266512 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00215/att-02/compact.done) = No such file or directory: 9.224 sec on http probe failed: 4, retry 2024/11/19-15:25:18.305974 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005195.sst): first pass time = 0.24's, 274.008'MB/sec 2024/11/19-15:25:18.306023 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005195.sst): old prealloc_size = 73819750, real_size = 55623440 2024/11/19-15:25:18.306259 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005195.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463564 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.77 avg-zval = 117.85 2024/11/19-15:25:18.306314 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005195.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.101 sec, Key+Value = 0.067 GB 2024/11/19-15:25:18.399337 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005195.sst): second pass time = 0.00's,241648214.286'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 595.012'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3753.635'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.643'MB/sec write SST data time = 0.05's, 1127.981'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 241648214.286'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463564 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 117.85 usrkeys = 463564 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6614 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5725 value = 0.9908 dict = 0.00 all = 1.2089 } Zip/UnZip{ index = 0.0946 value = 1.0093 dict = 0.00 all = 0.8272 } ---------------------------- total value len = 13.828211 GB avg = 0.250 KB (by entry num) total key len = 1.429208 GB avg = 0.026 KB total ukey len = 0.986534 GB avg = 0.018 KB total ukey num = 0.055334197 Billion total entry num = 0.055334197 Billion write speed all = 38.007558355 MB/sec (with seq num) write speed all = 36.904819918 MB/sec (without seq num) 2024/11/19-15:25:18.402912 139851158521600 [compaction_job.cc:1948] [default] [JOB 217] Generated table #5195: 463564 keys, 55512856 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:18.402971 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001118402948, "cf_name": "default", "job": 217, "event": "table_file_creation", "file_number": 5195, "file_size": 55512856, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54630464, "index_size": 876928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12979792, "raw_average_key_size": 28, "raw_value_size": 54129200, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463564, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001118, "oldest_key_time": 0, "file_creation_time": 1732001118, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5195, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:18.416661 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005196.sst) /storage/mytopling/..//tmp/Topling-82038-vZih3W 2024/11/19-15:25:18.479365 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005196.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0018 GB, waited 0.000 sec, Key+Value = 0.030 GB 2024/11/19-15:25:18.486240 139851148035840 [dcompact_etcd.cc:1163] job-00223/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.287 sec, input = 126.826 MiB 2024/11/19-15:25:18.493950 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005196.sst): index pass time = 0.01's, 224.051'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 203880 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 7 raw-key = 0.0033 GB zip-key = 0.0004 GB avg-key = 16.00 avg-zkey = 1.91 2024/11/19-15:25:18.524376 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005196.sst): first pass time = 0.11's, 274.031'MB/sec 2024/11/19-15:25:18.524426 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005196.sst): old prealloc_size = 73819750, real_size = 24459024 2024/11/19-15:25:18.524544 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005196.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 203880 prefix = 4 raw-key = 0.0033 GB zip-key = 0.0004 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0238 GB zip-val = 0.0240 GB avg-val = 116.76 avg-zval = 117.81 2024/11/19-15:25:18.524607 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005196.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0005 GB, waited 0.045 sec, Key+Value = 0.030 GB 2024/11/19-15:25:18.565765 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005196.sst): second pass time = 0.00's,147853614.907'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.04's, 590.721'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3469.144'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 2.627'MB/sec write SST data time = 0.02's, 1119.254'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.024'GB zip my value throughput = 147853614.907'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 203880 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.76 avg-zval = 117.81 usrkeys = 203880 avg-key = 20.00 avg-zkey = 1.91 avg-val = 116.76 avg-zval = 117.81 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 2912 UnZipSize{ index = 0.0041 GB value = 0.0238 GB dict = 0.00 MB all = 0.0295 GB } __ZipSize{ index = 0.0004 GB value = 0.0240 GB dict = 0.00 MB all = 0.0244 GB } UnZip/Zip{ index = 10.4790 value = 0.9911 dict = 0.00 all = 1.2090 } Zip/UnZip{ index = 0.0954 value = 1.0090 dict = 0.00 all = 0.8271 } ---------------------------- total value len = 13.852015 GB avg = 0.249 KB (by entry num) total key len = 1.434916 GB avg = 0.026 KB total ukey len = 0.990612 GB avg = 0.018 KB total ukey num = 0.055538077 Billion total entry num = 0.055538077 Billion write speed all = 38.065296530 MB/sec (with seq num) write speed all = 36.958953708 MB/sec (without seq num) 2024/11/19-15:25:18.567475 139851158521600 [compaction_job.cc:1948] [default] [JOB 217] Generated table #5196: 203880 keys, 24410200 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:18.567526 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001118567503, "cf_name": "default", "job": 217, "event": "table_file_creation", "file_number": 5196, "file_size": 24410200, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 24018752, "index_size": 389120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 5708640, "raw_average_key_size": 28, "raw_value_size": 23804432, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 203880, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001118, "oldest_key_time": 0, "file_creation_time": 1732001118, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5196, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:18.573878 139851158521600 [compaction_job.cc:703] job-00217: subcompact[0], size: 0.079923 G, files: /storage/mytopling/.rocksdb [005195,005196] 2024/11/19-15:25:18.575003 139851158521600 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 8717231 2024/11/19-15:25:18.575143 139851158521600 (Original Log Time 2024/11/19-15:25:18.060823) EVENT_LOG_v1 {"time_micros": 1732001118060809, "job": 217, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5133, 5137, 5142], "score": 6.14863, "input_data_size": 96112905, "oldest_snapshot_seqno": 144814564} 2024/11/19-15:25:18.575153 139851158521600 (Original Log Time 2024/11/19-15:25:18.574638) [compaction_job.cc:2022] [default] [JOB 217] Compacted 3@1 files to L2 => 79923056 bytes 2024/11/19-15:25:18.575159 139851158521600 (Original Log Time 2024/11/19-15:25:18.574971) [compaction_job.cc:1224] [default] compacted to: files[5 105 69 55 0 0 0] max score 5.03 (155 files need compaction), MB/sec: 187.3 rd, 155.8 wr, level 2, files in(3, 0) out(2 +0 blob) MB in(91.7, 0.0 +0.0 blob) out(76.2 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 667444, records dropped: 0 output_compression: Snappy 2024/11/19-15:25:18.575183 139851158521600 (Original Log Time 2024/11/19-15:25:18.574997) EVENT_LOG_v1 {"time_micros": 1732001118574985, "cf": "default", "job": 217, "event": "compaction_finished", "compaction_time_micros": 513066, "compaction_time_cpu_micros": 511087, "output_level": 2, "num_output_files": 2, "total_output_size": 79923056, "num_input_records": 667444, "num_output_records": 667444, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 105, 69, 55, 0, 0, 0]} 2024/11/19-15:25:18.575320 139851158521600 [compaction_job.cc:2437] [default] [JOB 224] Compacting 4@1 files to L2, score 5.03, subcompactions 1 : 1 2024/11/19-15:25:18.575334 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 222 Base level 1, inputs: [5045(31MB) 5050(31MB) 5051(31MB) 5052(31MB)] 2024/11/19-15:25:18.584126 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-02 2024/11/19-15:25:18.584205 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00216/att-02/compact.done) = No such file or directory: 9.217 sec on http probe failed: 4, retry 2024/11/19-15:25:18.694990 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-01 2024/11/19-15:25:18.695068 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00219/att-01/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:25:18.864880 139851158521600 [dcompact_etcd.cc:1163] job-00224/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 127.000 MiB 2024/11/19-15:25:19.941741 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-02 2024/11/19-15:25:19.941830 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00218/att-02/compact.done) = No such file or directory: 8.377 sec on http probe failed: 4, retry 2024/11/19-15:25:20.176988 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-01 2024/11/19-15:25:20.177064 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00221/att-01/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:25:20.274742 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-02 2024/11/19-15:25:20.274820 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00215/att-02/compact.done) = No such file or directory: 11.232 sec on http probe failed: 5, giveup 2024/11/19-15:25:20.274839 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00215/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.553 (submit 312.230 ms wait 11.232 s), input 127.820 MiB, send shutdown 2024/11/19-15:25:20.280064 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00215/att-02 2024/11/19-15:25:20.280275 139850978166528 [dcompact_etcd.cc:1773] job-00215/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:20.282093 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00215) = success 2024/11/19-15:25:20.282102 139850978166528 [dcompact_etcd.cc:1773] job-00215/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:25:20.282120 139850978166528 (Original Log Time 2024/11/19-15:24:45.659786) EVENT_LOG_v1 {"time_micros": 1732001085659780, "job": 215, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5102, 5104, 5106, 5108], "score": 6.64793, "input_data_size": 133300236, "oldest_snapshot_seqno": -1} 2024/11/19-15:25:20.282127 139850978166528 [compaction_job.cc:2437] [default] [JOB 215] Compacting 4@1 files to L2, score 6.65, subcompactions 1 : 1 2024/11/19-15:25:20.282134 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 214 Base level 1, inputs: [5102(31MB) 5104(31MB) 5106(31MB) 5108(31MB)] 2024/11/19-15:25:20.282378 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005197.sst) /storage/mytopling/..//tmp/Topling-82038-HtoFdU 2024/11/19-15:25:20.423640 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005197.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:20.457930 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005197.sst): index pass time = 0.03's, 216.480'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463355 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:25:20.522761 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005197.sst): first pass time = 0.24's, 279.190'MB/sec 2024/11/19-15:25:20.522813 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005197.sst): old prealloc_size = 73819750, real_size = 55606096 2024/11/19-15:25:20.523064 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005197.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463355 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.83 avg-zval = 117.87 2024/11/19-15:25:20.523140 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005197.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.100 sec, Key+Value = 0.067 GB 2024/11/19-15:25:20.593250 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-02 2024/11/19-15:25:20.593326 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00216/att-02/compact.done) = No such file or directory: 11.222 sec on http probe failed: 5, giveup 2024/11/19-15:25:20.593344 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00216/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.822 sec, real 11.524 (submit 292.168 ms wait 11.222 s), input 127.822 MiB, send shutdown 2024/11/19-15:25:20.602588 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00216/att-02 2024/11/19-15:25:20.602793 139851179493120 [dcompact_etcd.cc:1773] job-00216/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:20.605029 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00216) = success 2024/11/19-15:25:20.605037 139851179493120 [dcompact_etcd.cc:1773] job-00216/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:25:20.605057 139851179493120 (Original Log Time 2024/11/19-15:24:45.967016) EVENT_LOG_v1 {"time_micros": 1732001085967007, "job": 216, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5113, 5118, 5123, 5128], "score": 6.39828, "input_data_size": 133300364, "oldest_snapshot_seqno": -1} 2024/11/19-15:25:20.605065 139851179493120 [compaction_job.cc:2437] [default] [JOB 216] Compacting 4@1 files to L2, score 6.40, subcompactions 1 : 1 2024/11/19-15:25:20.605075 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 214 Base level 1, inputs: [5113(31MB) 5118(31MB) 5123(31MB) 5128(31MB)] 2024/11/19-15:25:20.605331 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005198.sst) /storage/mytopling/..//tmp/Topling-82038-1SMIkl 2024/11/19-15:25:20.616121 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005197.sst): second pass time = 0.00's,142086692.913'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 594.826'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3539.735'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.664'MB/sec write SST data time = 0.05's, 1119.562'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 142086692.913'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463355 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.83 avg-zval = 117.87 usrkeys = 463355 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.83 avg-zval = 117.87 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6615 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5708 value = 0.9912 dict = 0.00 all = 1.2093 } Zip/UnZip{ index = 0.0946 value = 1.0088 dict = 0.00 all = 0.8269 } ---------------------------- total value len = 13.906150 GB avg = 0.248 KB (by entry num) total key len = 1.447890 GB avg = 0.026 KB total ukey len = 0.999879 GB avg = 0.018 KB total ukey num = 0.056001432 Billion total entry num = 0.056001432 Billion write speed all = 38.038197156 MB/sec (with seq num) write speed all = 36.928290737 MB/sec (without seq num) 2024/11/19-15:25:20.619435 139850978166528 [compaction_job.cc:1948] [default] [JOB 215] Generated table #5197: 463355 keys, 55495512 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:20.619480 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001120619458, "cf_name": "default", "job": 215, "event": "table_file_creation", "file_number": 5197, "file_size": 55495512, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54613376, "index_size": 876672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12973940, "raw_average_key_size": 28, "raw_value_size": 54135030, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463355, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001120, "oldest_key_time": 0, "file_creation_time": 1732001120, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5197, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:20.632993 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005199.sst) /storage/mytopling/..//tmp/Topling-82038-zf32MN 2024/11/19-15:25:20.704181 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-01 2024/11/19-15:25:20.704279 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00219/att-01/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:25:20.704297 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00219/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.878 sec, real 11.521 (submit 304.395 ms wait 11.207 s), input 126.878 MiB, send shutdown 2024/11/19-15:25:20.712060 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-01 2024/11/19-15:25:20.712280 139851169007360 [dcompact_etcd.cc:1773] job-00219/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:20.746690 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005198.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:20.773620 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005199.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.006 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:20.783382 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005198.sst): index pass time = 0.04's, 202.259'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463442 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:25:20.810968 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005199.sst): index pass time = 0.04's, 198.240'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462227 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:25:20.854555 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005198.sst): first pass time = 0.25's, 269.286'MB/sec 2024/11/19-15:25:20.854612 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005198.sst): old prealloc_size = 73819750, real_size = 55613296 2024/11/19-15:25:20.854842 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005198.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463442 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.81 avg-zval = 117.86 2024/11/19-15:25:20.854930 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005198.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.108 sec, Key+Value = 0.067 GB 2024/11/19-15:25:20.876747 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005199.sst): first pass time = 0.24's, 274.557'MB/sec 2024/11/19-15:25:20.876798 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005199.sst): old prealloc_size = 73819750, real_size = 55469776 2024/11/19-15:25:20.877039 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005199.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462227 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.78 avg-zval = 117.85 2024/11/19-15:25:20.877093 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005199.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.103 sec, Key+Value = 0.067 GB 2024/11/19-15:25:20.949858 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005198.sst): second pass time = 0.00's,92534222.222'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 582.802'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3889.482'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.585'MB/sec write SST data time = 0.05's, 1105.683'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 92534222.222'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463442 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 117.86 usrkeys = 463442 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6615 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5720 value = 0.9911 dict = 0.00 all = 1.2091 } Zip/UnZip{ index = 0.0946 value = 1.0090 dict = 0.00 all = 0.8271 } ---------------------------- total value len = 13.960283 GB avg = 0.247 KB (by entry num) total key len = 1.460866 GB avg = 0.026 KB total ukey len = 1.009147 GB avg = 0.018 KB total ukey num = 0.056464874 Billion total entry num = 0.056464874 Billion write speed all = 38.172892017 MB/sec (with seq num) write speed all = 37.054725020 MB/sec (without seq num) 2024/11/19-15:25:20.971592 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005199.sst): second pass time = 0.00's,183597448.980'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 584.079'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3671.547'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.599'MB/sec write SST data time = 0.05's, 1105.496'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 183597448.980'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462227 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.78 avg-zval = 117.85 usrkeys = 462227 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.78 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6596 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0554 GB } UnZip/Zip{ index = 10.5052 value = 0.9909 dict = 0.00 all = 1.2088 } Zip/UnZip{ index = 0.0952 value = 1.0092 dict = 0.00 all = 0.8272 } ---------------------------- total value len = 14.014261 GB avg = 0.246 KB (by entry num) total key len = 1.473809 GB avg = 0.026 KB total ukey len = 1.018392 GB avg = 0.018 KB total ukey num = 0.056927101 Billion total entry num = 0.056927101 Billion write speed all = 38.336480528 MB/sec (with seq num) write speed all = 37.209220755 MB/sec (without seq num) 2024/11/19-15:25:20.995754 139851179493120 [compaction_job.cc:1948] [default] [JOB 216] Generated table #5198: 463442 keys, 55502744 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:20.995809 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001120995784, "cf_name": "default", "job": 216, "event": "table_file_creation", "file_number": 5198, "file_size": 55502744, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54620512, "index_size": 876736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12976376, "raw_average_key_size": 28, "raw_value_size": 54132520, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463442, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001120, "oldest_key_time": 0, "file_creation_time": 1732001120, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5198, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:20.995882 139850978166528 [compaction_job.cc:1948] [default] [JOB 215] Generated table #5199: 462227 keys, 55359512 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:20.995936 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001120995906, "cf_name": "default", "job": 215, "event": "table_file_creation", "file_number": 5199, "file_size": 55359512, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54474048, "index_size": 880000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12942356, "raw_average_key_size": 28, "raw_value_size": 53977650, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462227, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001120, "oldest_key_time": 0, "file_creation_time": 1732001120, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5199, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:21.008708 139851169007360 [dcompact_etcd.cc:1163] job-00219/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.296 sec, input = 126.878 MiB 2024/11/19-15:25:21.010031 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005200.sst) /storage/mytopling/..//tmp/Topling-82038-0rm2P9 2024/11/19-15:25:21.010087 139850978166528 [compaction_job.cc:703] job-00215: subcompact[0], size: 0.110855 G, files: /storage/mytopling/.rocksdb [005197,005199] 2024/11/19-15:25:21.011325 139850978166528 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 8986836 2024/11/19-15:25:21.011431 139850978166528 (Original Log Time 2024/11/19-15:25:20.282162) EVENT_LOG_v1 {"time_micros": 1732001120282146, "job": 215, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5102, 5104, 5106, 5108], "score": 6.64793, "input_data_size": 133300236, "oldest_snapshot_seqno": -1} 2024/11/19-15:25:21.011437 139850978166528 (Original Log Time 2024/11/19-15:25:21.010959) [compaction_job.cc:2022] [default] [JOB 215] Compacted 4@1 files to L2 => 110855024 bytes 2024/11/19-15:25:21.011440 139850978166528 (Original Log Time 2024/11/19-15:25:21.011296) [compaction_job.cc:1224] [default] compacted to: files[5 101 71 55 0 0 0] max score 4.78 (153 files need compaction), MB/sec: 183.1 rd, 152.3 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925582, records dropped: 0 output_compression: Snappy 2024/11/19-15:25:21.011448 139850978166528 (Original Log Time 2024/11/19-15:25:21.011320) EVENT_LOG_v1 {"time_micros": 1732001121011306, "cf": "default", "job": 215, "event": "compaction_finished", "compaction_time_micros": 727936, "compaction_time_cpu_micros": 703831, "output_level": 2, "num_output_files": 2, "total_output_size": 110855024, "num_input_records": 925582, "num_output_records": 925582, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 101, 71, 55, 0, 0, 0]} 2024/11/19-15:25:21.011529 139850978166528 [compaction_job.cc:2437] [default] [JOB 225] Compacting 4@1 files to L2, score 4.78, subcompactions 1 : 1 2024/11/19-15:25:21.011538 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 223 Base level 1, inputs: [5053(31MB) 5055(31MB) 5056(31MB) 5057(31MB)] 2024/11/19-15:25:21.150933 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005200.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:21.199476 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005200.sst): index pass time = 0.05's, 152.433'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462095 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:25:21.250981 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005200.sst): first pass time = 0.24's, 277.762'MB/sec 2024/11/19-15:25:21.251035 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005200.sst): old prealloc_size = 73819750, real_size = 55453728 2024/11/19-15:25:21.251278 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005200.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462095 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.82 avg-zval = 117.86 2024/11/19-15:25:21.251339 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005200.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.100 sec, Key+Value = 0.067 GB 2024/11/19-15:25:21.326377 139850978166528 [dcompact_etcd.cc:1163] job-00225/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.314 sec, input = 126.866 MiB 2024/11/19-15:25:21.344725 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005200.sst): second pass time = 0.00's,117100715.835'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 590.771'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3614.329'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.612'MB/sec write SST data time = 0.05's, 1124.640'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 117100715.835'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462095 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.82 avg-zval = 117.86 usrkeys = 462095 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.82 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6597 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0553 GB } UnZip/Zip{ index = 10.5721 value = 0.9912 dict = 0.00 all = 1.2092 } Zip/UnZip{ index = 0.0946 value = 1.0089 dict = 0.00 all = 0.8270 } ---------------------------- total value len = 14.068244 GB avg = 0.245 KB (by entry num) total key len = 1.486747 GB avg = 0.026 KB total ukey len = 1.027634 GB avg = 0.018 KB total ukey num = 0.057389196 Billion total entry num = 0.057389196 Billion write speed all = 38.466600421 MB/sec (with seq num) write speed all = 37.331238940 MB/sec (without seq num) 2024/11/19-15:25:21.348147 139851179493120 [compaction_job.cc:1948] [default] [JOB 216] Generated table #5200: 462095 keys, 55343512 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:21.348205 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001121348181, "cf_name": "default", "job": 216, "event": "table_file_creation", "file_number": 5200, "file_size": 55343512, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54463824, "index_size": 874176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12938660, "raw_average_key_size": 28, "raw_value_size": 53983430, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462095, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001121, "oldest_key_time": 0, "file_creation_time": 1732001121, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5200, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:21.361708 139851179493120 [compaction_job.cc:703] job-00216: subcompact[0], size: 0.110846 G, files: /storage/mytopling/.rocksdb [005198,005200] 2024/11/19-15:25:21.363023 139851179493120 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 9264779 2024/11/19-15:25:21.363154 139851179493120 (Original Log Time 2024/11/19-15:25:20.605108) EVENT_LOG_v1 {"time_micros": 1732001120605088, "job": 216, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5113, 5118, 5123, 5128], "score": 6.39828, "input_data_size": 133300364, "oldest_snapshot_seqno": -1} 2024/11/19-15:25:21.363178 139851179493120 (Original Log Time 2024/11/19-15:25:21.362658) [compaction_job.cc:2022] [default] [JOB 216] Compacted 4@1 files to L2 => 110846256 bytes 2024/11/19-15:25:21.363181 139851179493120 (Original Log Time 2024/11/19-15:25:21.362993) [compaction_job.cc:1224] [default] compacted to: files[5 97 73 55 0 0 0] max score 4.53 (151 files need compaction), MB/sec: 176.2 rd, 146.5 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925537, records dropped: 0 output_compression: Snappy 2024/11/19-15:25:21.363191 139851179493120 (Original Log Time 2024/11/19-15:25:21.363016) EVENT_LOG_v1 {"time_micros": 1732001121363005, "cf": "default", "job": 216, "event": "compaction_finished", "compaction_time_micros": 756611, "compaction_time_cpu_micros": 711647, "output_level": 2, "num_output_files": 2, "total_output_size": 110846256, "num_input_records": 925537, "num_output_records": 925537, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 97, 73, 55, 0, 0, 0]} 2024/11/19-15:25:21.363865 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005024.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:21.363883 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001121363881, "job": 216, "event": "table_file_deletion", "file_number": 5024} 2024/11/19-15:25:21.366889 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005028.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:21.366910 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001121366908, "job": 216, "event": "table_file_deletion", "file_number": 5028} 2024/11/19-15:25:21.370694 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005034.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:21.370714 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001121370712, "job": 216, "event": "table_file_deletion", "file_number": 5034} 2024/11/19-15:25:21.374497 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005037.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:21.374516 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001121374514, "job": 216, "event": "table_file_deletion", "file_number": 5037} 2024/11/19-15:25:21.378705 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005038.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:21.378725 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001121378723, "job": 216, "event": "table_file_deletion", "file_number": 5038} 2024/11/19-15:25:21.383047 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005040.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:21.383065 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001121383063, "job": 216, "event": "table_file_deletion", "file_number": 5040} 2024/11/19-15:25:21.387268 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005042.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:21.387287 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001121387285, "job": 216, "event": "table_file_deletion", "file_number": 5042} 2024/11/19-15:25:21.391446 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005046.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:21.391464 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001121391462, "job": 216, "event": "table_file_deletion", "file_number": 5046} 2024/11/19-15:25:21.391592 139851179493120 [compaction_job.cc:2437] [default] [JOB 226] Compacting 4@1 files to L2, score 4.53, subcompactions 1 : 1 2024/11/19-15:25:21.391600 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 224 Base level 1, inputs: [5060(31MB) 5061(31MB) 5062(31MB) 5064(31MB)] 2024/11/19-15:25:21.667053 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-00 2024/11/19-15:25:21.667154 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00223/att-00/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:25:21.684544 139851179493120 [dcompact_etcd.cc:1163] job-00226/att-00: init time wrt = 0.001, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.292 sec, input = 126.998 MiB 2024/11/19-15:25:21.912813 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-02 2024/11/19-15:25:21.912883 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00218/att-02/compact.done) = No such file or directory: 10.351 sec on http probe failed: 5, giveup 2024/11/19-15:25:21.912901 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00218/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 98.280 sec, real 10.629 (submit 272.554 ms wait 10.351 s), input 98.280 MiB, send shutdown 2024/11/19-15:25:21.918640 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00218/att-02 2024/11/19-15:25:21.918839 139851127064320 [dcompact_etcd.cc:1773] job-00218/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:21.920591 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00218) = success 2024/11/19-15:25:21.920599 139851127064320 [dcompact_etcd.cc:1773] job-00218/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:25:21.920616 139851127064320 (Original Log Time 2024/11/19-15:24:49.952475) EVENT_LOG_v1 {"time_micros": 1732001089952467, "job": 218, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5146, 5148, 5150, 5017], "score": 5.96866, "input_data_size": 103883519, "oldest_snapshot_seqno": -1} 2024/11/19-15:25:21.920624 139851127064320 [compaction_job.cc:2437] [default] [JOB 218] Compacting 4@1 files to L2, score 5.97, subcompactions 1 : 1 2024/11/19-15:25:21.920633 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 216 Base level 1, inputs: [5146(31MB) 5148(31MB) 5150(3231KB) 5017(31MB)] 2024/11/19-15:25:21.920824 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005201.sst) /storage/mytopling/..//tmp/Topling-82038-zwbF5b 2024/11/19-15:25:21.974438 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005201.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:21.977741 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005201.sst): index pass time = 0.00's, 158.367'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64715 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:25:22.042428 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005201.sst): first pass time = 0.12's, 551.933'MB/sec 2024/11/19-15:25:22.042478 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005201.sst): old prealloc_size = 73819750, real_size = 66152208 2024/11/19-15:25:22.042535 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005201.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64715 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.00 avg-zval =1020.25 2024/11/19-15:25:22.042580 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005201.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.068 sec, Key+Value = 0.067 GB 2024/11/19-15:25:22.046296 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-00 2024/11/19-15:25:22.046395 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00224/att-00/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:25:22.104231 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005201.sst): second pass time = 0.00's,203762006.192'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1071.338'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2087.115'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.367'MB/sec write SST data time = 0.02's, 3751.955'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 203762006.192'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64715 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.25 usrkeys = 64715 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8063 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0465 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 14.134059 GB avg = 0.246 KB (by entry num) total key len = 1.488042 GB avg = 0.026 KB total ukey len = 1.028410 GB avg = 0.018 KB total ukey num = 0.057453911 Billion total entry num = 0.057453911 Billion write speed all = 38.560134555 MB/sec (with seq num) write speed all = 37.425623620 MB/sec (without seq num) 2024/11/19-15:25:22.182862 139851127064320 [compaction_job.cc:1948] [default] [JOB 218] Generated table #5201: 64715 keys, 66137412 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:22.182929 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001122182894, "cf_name": "default", "job": 218, "event": "table_file_creation", "file_number": 5201, "file_size": 66137412, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025664, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294300, "raw_average_key_size": 20, "raw_value_size": 65815128, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64715, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001121, "oldest_key_time": 0, "file_creation_time": 1732001121, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5201, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:22.186525 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-01 2024/11/19-15:25:22.186599 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00221/att-01/compact.done) = No such file or directory: 9.194 sec on http probe failed: 4, retry 2024/11/19-15:25:22.197905 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005202.sst) /storage/mytopling/..//tmp/Topling-82038-JPi3DB 2024/11/19-15:25:22.226823 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005202.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.036 GB 2024/11/19-15:25:22.226934 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005202.sst): index pass time = 0.00's,3445.056'MB/sec index type = UintIndex_AllOne usrkeys = 34662 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0003 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:25:22.262413 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005202.sst): first pass time = 0.06's, 557.350'MB/sec 2024/11/19-15:25:22.262465 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005202.sst): old prealloc_size = 73819750, real_size = 35373032 2024/11/19-15:25:22.262490 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005202.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 34662 prefix = 4 raw-key = 0.0003 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0353 GB zip-val = 0.0354 GB avg-val =1017.00 avg-zval =1020.25 2024/11/19-15:25:22.291116 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005202.sst): second pass time = 0.00's,164724990.654'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.03's, 1234.991'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.434'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 0.431'MB/sec write SST data time = 0.01's, 4157.879'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.035'GB zip my value throughput = 164724990.654'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 34662 avg-key = 20.00 avg-zkey = 0.01 avg-val = 1017.00 avg-zval = 1020.25 usrkeys = 34662 avg-key = 12.00 avg-zkey = 0.01 avg-val = 1017.00 avg-zval = 1020.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0004 GB value = 0.0353 GB dict = 0.00 MB all = 0.0359 GB } __ZipSize{ index = 0.0000 GB value = 0.0354 GB dict = 0.00 MB all = 0.0354 GB } UnZip/Zip{ index =2166.3750 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0005 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 14.169310 GB avg = 0.246 KB (by entry num) total key len = 1.488735 GB avg = 0.026 KB total ukey len = 1.028826 GB avg = 0.018 KB total ukey num = 0.057488573 Billion total entry num = 0.057488573 Billion write speed all = 38.631036204 MB/sec (with seq num) write speed all = 37.496364231 MB/sec (without seq num) 2024/11/19-15:25:22.300437 139851127064320 [compaction_job.cc:1948] [default] [JOB 218] Generated table #5202: 34662 keys, 35365322 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:22.300488 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001122300465, "cf_name": "default", "job": 218, "event": "table_file_creation", "file_number": 5202, "file_size": 35365322, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 35364016, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 693240, "raw_average_key_size": 20, "raw_value_size": 35251148, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 34662, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001122, "oldest_key_time": 0, "file_creation_time": 1732001122, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5202, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:22.308393 139851127064320 [compaction_job.cc:703] job-00218: subcompact[0], size: 0.101503 G, files: /storage/mytopling/.rocksdb [005201,005202] 2024/11/19-15:25:22.314314 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 9551318 2024/11/19-15:25:22.314465 139851127064320 (Original Log Time 2024/11/19-15:25:21.920660) EVENT_LOG_v1 {"time_micros": 1732001121920644, "job": 218, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5146, 5148, 5150, 5017], "score": 5.96866, "input_data_size": 103883519, "oldest_snapshot_seqno": -1} 2024/11/19-15:25:22.314475 139851127064320 (Original Log Time 2024/11/19-15:25:22.313952) [compaction_job.cc:2022] [default] [JOB 218] Compacted 4@1 files to L2 => 101502734 bytes 2024/11/19-15:25:22.314481 139851127064320 (Original Log Time 2024/11/19-15:25:22.314284) [compaction_job.cc:1224] [default] compacted to: files[5 93 75 55 0 0 0] max score 4.28 (149 files need compaction), MB/sec: 267.9 rd, 261.8 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(99.1, 0.0 +0.0 blob) out(96.8 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 99377, records dropped: 0 output_compression: Snappy 2024/11/19-15:25:22.314496 139851127064320 (Original Log Time 2024/11/19-15:25:22.314308) EVENT_LOG_v1 {"time_micros": 1732001122314296, "cf": "default", "job": 218, "event": "compaction_finished", "compaction_time_micros": 387746, "compaction_time_cpu_micros": 303121, "output_level": 2, "num_output_files": 2, "total_output_size": 101502734, "num_input_records": 99377, "num_output_records": 99377, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 93, 75, 55, 0, 0, 0]} 2024/11/19-15:25:22.330812 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005054.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:22.330838 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001122330835, "job": 218, "event": "table_file_deletion", "file_number": 5054} 2024/11/19-15:25:22.351995 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005063.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:22.352020 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001122352017, "job": 218, "event": "table_file_deletion", "file_number": 5063} 2024/11/19-15:25:22.382902 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005067.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:22.382934 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001122382932, "job": 218, "event": "table_file_deletion", "file_number": 5067} 2024/11/19-15:25:22.411991 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005072.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:22.412014 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001122412011, "job": 218, "event": "table_file_deletion", "file_number": 5072} 2024/11/19-15:25:22.412160 139851127064320 [compaction_job.cc:2437] [default] [JOB 227] Compacting 4@1 files to L2, score 4.28, subcompactions 1 : 1 2024/11/19-15:25:22.412170 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 225 Base level 1, inputs: [5065(31MB) 5066(31MB) 5068(31MB) 5069(31MB)] 2024/11/19-15:25:22.740181 139851127064320 [dcompact_etcd.cc:1163] job-00227/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.328 sec, input = 126.878 MiB 2024/11/19-15:25:23.672655 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-00 2024/11/19-15:25:23.672734 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00223/att-00/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:25:24.056396 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-00 2024/11/19-15:25:24.056474 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00224/att-00/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:25:24.189656 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-02 2024/11/19-15:25:24.189766 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00219/att-02/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:25:24.195249 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-01 2024/11/19-15:25:24.195328 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00221/att-01/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:25:24.195352 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00221/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.908 sec, real 11.509 (submit 296.844 ms wait 11.204 s), input 126.908 MiB, send shutdown 2024/11/19-15:25:24.203249 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-01 2024/11/19-15:25:24.203489 139851137550080 [dcompact_etcd.cc:1773] job-00221/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:24.502777 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-00 2024/11/19-15:25:24.502859 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00225/att-00/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:25:24.510220 139851137550080 [dcompact_etcd.cc:1163] job-00221/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.307 sec, input = 126.908 MiB 2024/11/19-15:25:24.866148 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-00 2024/11/19-15:25:24.866255 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00226/att-00/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:25:25.682371 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-00 2024/11/19-15:25:25.682476 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00223/att-00/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:25:25.918031 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-00 2024/11/19-15:25:25.918117 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00227/att-00/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:25:26.065068 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-00 2024/11/19-15:25:26.065130 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00224/att-00/compact.done) = No such file or directory: 7.192 sec on http probe failed: 3, retry 2024/11/19-15:25:26.194590 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-02 2024/11/19-15:25:26.194664 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00219/att-02/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:25:26.511370 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-00 2024/11/19-15:25:26.511464 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00225/att-00/compact.done) = No such file or directory: 5.177 sec on http probe failed: 2, retry 2024/11/19-15:25:26.874426 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-00 2024/11/19-15:25:26.874529 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00226/att-00/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:25:27.691159 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-00 2024/11/19-15:25:27.691229 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00223/att-00/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:25:27.692526 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-02 2024/11/19-15:25:27.692598 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00221/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:25:27.927552 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-00 2024/11/19-15:25:27.927633 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00227/att-00/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:25:28.073953 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-00 2024/11/19-15:25:28.074036 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00224/att-00/compact.done) = No such file or directory: 9.200 sec on http probe failed: 4, retry 2024/11/19-15:25:28.199634 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-02 2024/11/19-15:25:28.199720 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00219/att-02/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:25:28.521237 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-00 2024/11/19-15:25:28.521315 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00225/att-00/compact.done) = No such file or directory: 7.185 sec on http probe failed: 3, retry 2024/11/19-15:25:28.883063 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-00 2024/11/19-15:25:28.883153 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00226/att-00/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:25:29.700608 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-00 2024/11/19-15:25:29.700708 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00223/att-00/compact.done) = No such file or directory: 11.205 sec on http probe failed: 5, giveup 2024/11/19-15:25:29.700727 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00223/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.826 sec, real 11.501 (submit 286.801 ms wait 11.205 s), input 126.826 MiB, send shutdown 2024/11/19-15:25:29.701057 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-02 2024/11/19-15:25:29.701128 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00221/att-02/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:25:29.709125 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-00 2024/11/19-15:25:29.709338 139851148035840 [dcompact_etcd.cc:1773] job-00223/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:29.932423 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-00 2024/11/19-15:25:29.932496 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00227/att-00/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:25:30.008062 139851148035840 [dcompact_etcd.cc:1163] job-00223/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.299 sec, input = 126.826 MiB 2024/11/19-15:25:30.083879 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-00 2024/11/19-15:25:30.083960 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00224/att-00/compact.done) = No such file or directory: 11.209 sec on http probe failed: 5, giveup 2024/11/19-15:25:30.083978 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00224/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.000 sec, real 11.508 (submit 289.094 ms wait 11.209 s), input 127.000 MiB, send shutdown 2024/11/19-15:25:30.091409 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-00 2024/11/19-15:25:30.091617 139851158521600 [dcompact_etcd.cc:1773] job-00224/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:30.207887 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-02 2024/11/19-15:25:30.207955 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00219/att-02/compact.done) = No such file or directory: 9.191 sec on http probe failed: 4, retry 2024/11/19-15:25:30.354078 139851158521600 [dcompact_etcd.cc:1163] job-00224/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.262 sec, input = 127.000 MiB 2024/11/19-15:25:30.529228 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-00 2024/11/19-15:25:30.529304 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00225/att-00/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:25:30.888286 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-00 2024/11/19-15:25:30.888352 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00226/att-00/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:25:31.711098 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-02 2024/11/19-15:25:31.711199 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00221/att-02/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:25:31.940730 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-00 2024/11/19-15:25:31.940803 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00227/att-00/compact.done) = No such file or directory: 9.192 sec on http probe failed: 4, retry 2024/11/19-15:25:32.216487 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-02 2024/11/19-15:25:32.216563 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00219/att-02/compact.done) = No such file or directory: 11.199 sec on http probe failed: 5, giveup 2024/11/19-15:25:32.216582 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00219/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.878 sec, real 11.504 (submit 296.358 ms wait 11.199 s), input 126.878 MiB, send shutdown 2024/11/19-15:25:32.223598 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00219/att-02 2024/11/19-15:25:32.223804 139851169007360 [dcompact_etcd.cc:1773] job-00219/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:32.225897 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00219) = success 2024/11/19-15:25:32.225904 139851169007360 [dcompact_etcd.cc:1773] job-00219/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:25:32.225932 139851169007360 (Original Log Time 2024/11/19-15:24:57.684471) EVENT_LOG_v1 {"time_micros": 1732001097684464, "job": 219, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5020, 5022, 5025, 5027], "score": 5.77516, "input_data_size": 134098368, "oldest_snapshot_seqno": 145642088} 2024/11/19-15:25:32.225939 139851169007360 [compaction_job.cc:2437] [default] [JOB 219] Compacting 4@1 files to L2, score 5.78, subcompactions 1 : 1 2024/11/19-15:25:32.225951 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 217 Base level 1, inputs: [5020(31MB) 5022(31MB) 5025(31MB) 5027(31MB)] 2024/11/19-15:25:32.226178 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005203.sst) /storage/mytopling/..//tmp/Topling-82038-S5N1iP 2024/11/19-15:25:32.278980 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005203.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:32.284312 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005203.sst): index pass time = 0.01's, 97.760'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64713 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:25:32.345538 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005203.sst): first pass time = 0.12's, 562.302'MB/sec 2024/11/19-15:25:32.345590 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005203.sst): old prealloc_size = 73819750, real_size = 66151696 2024/11/19-15:25:32.345647 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005203.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64713 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.02 avg-zval =1020.28 2024/11/19-15:25:32.345707 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005203.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.067 GB 2024/11/19-15:25:32.404482 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005203.sst): second pass time = 0.00's,216495473.684'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1123.355'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2031.222'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.384'MB/sec write SST data time = 0.02's, 3960.532'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 216495473.684'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64713 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.02 avg-zval = 1020.28 usrkeys = 64713 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.02 avg-zval = 1020.28 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8063 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0463 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 14.235125 GB avg = 0.247 KB (by entry num) total key len = 1.490029 GB avg = 0.026 KB total ukey len = 1.029603 GB avg = 0.018 KB total ukey num = 0.057553286 Billion total entry num = 0.057553286 Billion write speed all = 37.852141977 MB/sec (with seq num) write speed all = 36.743846248 MB/sec (without seq num) 2024/11/19-15:25:32.433887 139851169007360 [compaction_job.cc:1948] [default] [JOB 219] Generated table #5203: 64713 keys, 66136900 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:32.433957 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001132433916, "cf_name": "default", "job": 219, "event": "table_file_creation", "file_number": 5203, "file_size": 66136900, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025152, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294260, "raw_average_key_size": 20, "raw_value_size": 65814624, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64713, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001132, "oldest_key_time": 0, "file_creation_time": 1732001132, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5203, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:32.448093 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005204.sst) /storage/mytopling/..//tmp/Topling-82038-mIrSOr 2024/11/19-15:25:32.505184 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005204.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:25:32.510402 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005204.sst): index pass time = 0.01's, 98.216'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63582 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:25:32.538622 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-00 2024/11/19-15:25:32.538720 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00225/att-00/compact.done) = No such file or directory: 11.203 sec on http probe failed: 5, giveup 2024/11/19-15:25:32.538738 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00225/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.866 sec, real 11.527 (submit 314.381 ms wait 11.203 s), input 126.866 MiB, send shutdown 2024/11/19-15:25:32.546072 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-00 2024/11/19-15:25:32.546292 139850978166528 [dcompact_etcd.cc:1773] job-00225/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:32.571117 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005204.sst): first pass time = 0.12's, 536.008'MB/sec 2024/11/19-15:25:32.571162 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005204.sst): old prealloc_size = 73819750, real_size = 64992432 2024/11/19-15:25:32.571219 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005204.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63582 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1016.97 avg-zval =1020.23 2024/11/19-15:25:32.571261 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005204.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.066 GB 2024/11/19-15:25:32.629039 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005204.sst): second pass time = 0.00's,284851140.969'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1123.052'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2040.782'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.385'MB/sec write SST data time = 0.02's, 3938.591'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 284851140.969'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63582 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.97 avg-zval = 1020.23 usrkeys = 63582 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.97 avg-zval = 1020.23 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7923 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0459 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 14.299786 GB avg = 0.248 KB (by entry num) total key len = 1.491301 GB avg = 0.026 KB total ukey len = 1.030366 GB avg = 0.018 KB total ukey num = 0.057616868 Billion total entry num = 0.057616868 Billion write speed all = 37.990314423 MB/sec (with seq num) write speed all = 36.881393712 MB/sec (without seq num) 2024/11/19-15:25:32.632967 139851169007360 [compaction_job.cc:1948] [default] [JOB 219] Generated table #5204: 63582 keys, 64977924 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:32.633019 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001132632996, "cf_name": "default", "job": 219, "event": "table_file_creation", "file_number": 5204, "file_size": 64977924, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64868064, "index_size": 108288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1271640, "raw_average_key_size": 20, "raw_value_size": 64661209, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63582, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001132, "oldest_key_time": 0, "file_creation_time": 1732001132, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5204, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:32.646843 139851169007360 [compaction_job.cc:703] job-00219: subcompact[0], size: 0.131115 G, files: /storage/mytopling/.rocksdb [005203,005204] 2024/11/19-15:25:32.647675 139851169007360 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 9846719 2024/11/19-15:25:32.647807 139851169007360 (Original Log Time 2024/11/19-15:25:32.225979) EVENT_LOG_v1 {"time_micros": 1732001132225964, "job": 219, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5020, 5022, 5025, 5027], "score": 5.77516, "input_data_size": 134098368, "oldest_snapshot_seqno": 145642088} 2024/11/19-15:25:32.647814 139851169007360 (Original Log Time 2024/11/19-15:25:32.647325) [compaction_job.cc:2022] [default] [JOB 219] Compacted 4@1 files to L2 => 131114824 bytes 2024/11/19-15:25:32.647817 139851169007360 (Original Log Time 2024/11/19-15:25:32.647643) [compaction_job.cc:1224] [default] compacted to: files[5 89 77 55 0 0 0] max score 4.03 (147 files need compaction), MB/sec: 318.6 rd, 311.5 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128295, records dropped: 0 output_compression: Snappy 2024/11/19-15:25:32.647835 139851169007360 (Original Log Time 2024/11/19-15:25:32.647668) EVENT_LOG_v1 {"time_micros": 1732001132647656, "cf": "default", "job": 219, "event": "compaction_finished", "compaction_time_micros": 420876, "compaction_time_cpu_micros": 390896, "output_level": 2, "num_output_files": 2, "total_output_size": 131114824, "num_input_records": 128295, "num_output_records": 128295, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 89, 77, 55, 0, 0, 0]} 2024/11/19-15:25:32.651817 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005076.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:32.651842 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001132651839, "job": 219, "event": "table_file_deletion", "file_number": 5076} 2024/11/19-15:25:32.656218 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005080.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:32.656240 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001132656238, "job": 219, "event": "table_file_deletion", "file_number": 5080} 2024/11/19-15:25:32.660377 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005084.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:32.660397 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001132660395, "job": 219, "event": "table_file_deletion", "file_number": 5084} 2024/11/19-15:25:32.664535 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005088.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:32.664554 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001132664552, "job": 219, "event": "table_file_deletion", "file_number": 5088} 2024/11/19-15:25:32.664681 139851169007360 [compaction_job.cc:2437] [default] [JOB 228] Compacting 4@1 files to L2, score 4.03, subcompactions 1 : 1 2024/11/19-15:25:32.664690 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 226 Base level 1, inputs: [5070(31MB) 5071(31MB) 5073(31MB) 5074(31MB)] 2024/11/19-15:25:32.825516 139850978166528 [dcompact_etcd.cc:1163] job-00225/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.279 sec, input = 126.866 MiB 2024/11/19-15:25:32.896601 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-00 2024/11/19-15:25:32.896660 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00226/att-00/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:25:32.896676 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00226/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.998 sec, real 11.504 (submit 292.226 ms wait 11.204 s), input 126.998 MiB, send shutdown 2024/11/19-15:25:32.905242 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-00 2024/11/19-15:25:32.905448 139851179493120 [dcompact_etcd.cc:1773] job-00226/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:32.961316 139851169007360 [dcompact_etcd.cc:1163] job-00228/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.296 sec, input = 126.892 MiB 2024/11/19-15:25:33.184990 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-01 2024/11/19-15:25:33.185067 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00223/att-01/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:25:33.185846 139851179493120 [dcompact_etcd.cc:1163] job-00226/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 126.998 MiB 2024/11/19-15:25:33.538750 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-01 2024/11/19-15:25:33.538832 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00224/att-01/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:25:33.719698 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-02 2024/11/19-15:25:33.719755 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00221/att-02/compact.done) = No such file or directory: 9.201 sec on http probe failed: 4, retry 2024/11/19-15:25:33.948814 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-00 2024/11/19-15:25:33.948874 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00227/att-00/compact.done) = No such file or directory: 11.201 sec on http probe failed: 5, giveup 2024/11/19-15:25:33.948892 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00227/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.878 sec, real 11.536 (submit 327.559 ms wait 11.201 s), input 126.878 MiB, send shutdown 2024/11/19-15:25:33.953545 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-00 2024/11/19-15:25:33.953748 139851127064320 [dcompact_etcd.cc:1773] job-00227/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:34.236077 139851127064320 [dcompact_etcd.cc:1163] job-00227/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.282 sec, input = 126.878 MiB 2024/11/19-15:25:35.193221 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-01 2024/11/19-15:25:35.193297 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00223/att-01/compact.done) = No such file or directory: 5.177 sec on http probe failed: 2, retry 2024/11/19-15:25:35.318307 139852362282752 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 9551317 2024/11/19-15:25:35.544157 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-01 2024/11/19-15:25:35.544254 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00224/att-01/compact.done) = No such file or directory: 5.185 sec on http probe failed: 2, retry 2024/11/19-15:25:35.728179 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-02 2024/11/19-15:25:35.728249 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00221/att-02/compact.done) = No such file or directory: 11.210 sec on http probe failed: 5, giveup 2024/11/19-15:25:35.728267 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00221/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.908 sec, real 11.525 (submit 306.629 ms wait 11.210 s), input 126.908 MiB, send shutdown 2024/11/19-15:25:35.737173 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00221/att-02 2024/11/19-15:25:35.737381 139851137550080 [dcompact_etcd.cc:1773] job-00221/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:35.739227 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00221) = success 2024/11/19-15:25:35.739242 139851137550080 [dcompact_etcd.cc:1773] job-00221/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:25:35.739268 139851137550080 (Original Log Time 2024/11/19-15:25:01.157029) EVENT_LOG_v1 {"time_micros": 1732001101157022, "job": 221, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5029, 5031, 5033, 5035], "score": 5.52538, "input_data_size": 134099136, "oldest_snapshot_seqno": -1} 2024/11/19-15:25:35.739275 139851137550080 [compaction_job.cc:2437] [default] [JOB 221] Compacting 4@1 files to L2, score 5.53, subcompactions 1 : 1 2024/11/19-15:25:35.739283 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 219 Base level 1, inputs: [5029(31MB) 5031(31MB) 5033(31MB) 5035(31MB)] 2024/11/19-15:25:35.739523 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005205.sst) /storage/mytopling/..//tmp/Topling-82038-cfxHGs 2024/11/19-15:25:35.791523 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005205.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:35.791654 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005205.sst): index pass time = 0.00's,5048.809'MB/sec index type = UintIndex_AllOne usrkeys = 64715 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:25:35.857996 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005205.sst): first pass time = 0.12's, 566.516'MB/sec 2024/11/19-15:25:35.858059 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005205.sst): old prealloc_size = 73819750, real_size = 66041624 2024/11/19-15:25:35.858087 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005205.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64715 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.99 avg-zval =1020.24 2024/11/19-15:25:35.912720 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005205.sst): second pass time = 0.00's,369744870.787'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1208.163'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 8.968'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.414'MB/sec write SST data time = 0.02's, 4243.518'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 369744870.787'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64715 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.99 avg-zval = 1020.24 usrkeys = 64715 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.99 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.6875 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 14.365601 GB avg = 0.249 KB (by entry num) total key len = 1.492595 GB avg = 0.026 KB total ukey len = 1.031143 GB avg = 0.018 KB total ukey num = 0.057681583 Billion total entry num = 0.057681583 Billion write speed all = 37.852732484 MB/sec (with seq num) write speed all = 36.751267706 MB/sec (without seq num) 2024/11/19-15:25:35.930213 139851137550080 [compaction_job.cc:1948] [default] [JOB 221] Generated table #5205: 64715 keys, 66026442 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:35.930266 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001135930242, "cf_name": "default", "job": 221, "event": "table_file_creation", "file_number": 5205, "file_size": 66026442, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025120, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294300, "raw_average_key_size": 20, "raw_value_size": 65814587, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64715, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001135, "oldest_key_time": 0, "file_creation_time": 1732001135, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5205, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:35.944658 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005206.sst) /storage/mytopling/..//tmp/Topling-82038-mhSRHu 2024/11/19-15:25:35.997579 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005206.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:25:36.000837 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005206.sst): index pass time = 0.00's, 157.714'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63608 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:25:36.006037 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-01 2024/11/19-15:25:36.006115 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00225/att-01/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:25:36.071695 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005206.sst): first pass time = 0.13's, 519.315'MB/sec 2024/11/19-15:25:36.071746 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005206.sst): old prealloc_size = 73819750, real_size = 65022496 2024/11/19-15:25:36.071804 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005206.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63608 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1017.03 avg-zval =1020.28 2024/11/19-15:25:36.071871 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005206.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.074 sec, Key+Value = 0.066 GB 2024/11/19-15:25:36.130987 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005206.sst): second pass time = 0.00's,412045299.363'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1097.678'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1994.882'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.373'MB/sec write SST data time = 0.02's, 3924.817'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 412045299.363'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63608 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 1020.28 usrkeys = 63608 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 1020.28 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7927 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0446 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 14.430292 GB avg = 0.250 KB (by entry num) total key len = 1.493867 GB avg = 0.026 KB total ukey len = 1.031906 GB avg = 0.018 KB total ukey num = 0.057745191 Billion total entry num = 0.057745191 Billion write speed all = 37.990390909 MB/sec (with seq num) write speed all = 36.888285685 MB/sec (without seq num) 2024/11/19-15:25:36.134394 139851137550080 [compaction_job.cc:1948] [default] [JOB 221] Generated table #5206: 63608 keys, 65008004 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:36.134445 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001136134421, "cf_name": "default", "job": 221, "event": "table_file_creation", "file_number": 5206, "file_size": 65008004, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64898064, "index_size": 108352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1272160, "raw_average_key_size": 20, "raw_value_size": 64691112, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63608, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001135, "oldest_key_time": 0, "file_creation_time": 1732001135, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5206, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:36.143600 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-00 2024/11/19-15:25:36.143674 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00228/att-00/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:25:36.148303 139851137550080 [compaction_job.cc:703] job-00221: subcompact[0], size: 0.131034 G, files: /storage/mytopling/.rocksdb [005205,005206] 2024/11/19-15:25:36.149038 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 9846718 2024/11/19-15:25:36.149175 139851137550080 (Original Log Time 2024/11/19-15:25:35.739311) EVENT_LOG_v1 {"time_micros": 1732001135739296, "job": 221, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5029, 5031, 5033, 5035], "score": 5.52538, "input_data_size": 134099136, "oldest_snapshot_seqno": -1} 2024/11/19-15:25:36.149181 139851137550080 (Original Log Time 2024/11/19-15:25:36.148669) [compaction_job.cc:2022] [default] [JOB 221] Compacted 4@1 files to L2 => 131034446 bytes 2024/11/19-15:25:36.149184 139851137550080 (Original Log Time 2024/11/19-15:25:36.149008) [compaction_job.cc:1224] [default] compacted to: files[5 85 79 55 0 0 0] max score 3.78 (145 files need compaction), MB/sec: 327.9 rd, 320.4 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128323, records dropped: 0 output_compression: Snappy 2024/11/19-15:25:36.149201 139851137550080 (Original Log Time 2024/11/19-15:25:36.149031) EVENT_LOG_v1 {"time_micros": 1732001136149019, "cf": "default", "job": 221, "event": "compaction_finished", "compaction_time_micros": 409005, "compaction_time_cpu_micros": 390093, "output_level": 2, "num_output_files": 2, "total_output_size": 131034446, "num_input_records": 128323, "num_output_records": 128323, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 85, 79, 55, 0, 0, 0]} 2024/11/19-15:25:36.153175 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005091.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:36.153197 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001136153195, "job": 221, "event": "table_file_deletion", "file_number": 5091} 2024/11/19-15:25:36.157139 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005095.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:36.157160 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001136157158, "job": 221, "event": "table_file_deletion", "file_number": 5095} 2024/11/19-15:25:36.161271 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005098.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:36.161290 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001136161288, "job": 221, "event": "table_file_deletion", "file_number": 5098} 2024/11/19-15:25:36.165226 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005100.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:36.165246 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001136165244, "job": 221, "event": "table_file_deletion", "file_number": 5100} 2024/11/19-15:25:36.165371 139851137550080 [compaction_job.cc:2437] [default] [JOB 229] Compacting 4@1 files to L2, score 3.78, subcompactions 1 : 1 2024/11/19-15:25:36.165380 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 227 Base level 1, inputs: [5075(31MB) 5077(31MB) 5078(31MB) 5079(31MB)] 2024/11/19-15:25:36.369515 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-01 2024/11/19-15:25:36.369593 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00226/att-01/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:25:36.449227 139851137550080 [dcompact_etcd.cc:1163] job-00229/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.283 sec, input = 126.980 MiB 2024/11/19-15:25:37.201719 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-01 2024/11/19-15:25:37.201792 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00223/att-01/compact.done) = No such file or directory: 7.185 sec on http probe failed: 3, retry 2024/11/19-15:25:37.414198 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-01 2024/11/19-15:25:37.414268 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00227/att-01/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:25:37.552304 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-01 2024/11/19-15:25:37.552362 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00224/att-01/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:25:38.014719 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-01 2024/11/19-15:25:38.014797 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00225/att-01/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:25:38.166468 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-00 2024/11/19-15:25:38.166535 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00228/att-00/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:25:38.376215 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-01 2024/11/19-15:25:38.376287 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00226/att-01/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:25:39.207462 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-01 2024/11/19-15:25:39.207548 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00223/att-01/compact.done) = No such file or directory: 9.194 sec on http probe failed: 4, retry 2024/11/19-15:25:39.419790 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-01 2024/11/19-15:25:39.419874 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00227/att-01/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:25:39.560767 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-01 2024/11/19-15:25:39.560874 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00224/att-01/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:25:39.636551 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-00 2024/11/19-15:25:39.636653 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00229/att-00/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:25:40.022858 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-01 2024/11/19-15:25:40.022942 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00225/att-01/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:25:40.175610 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-00 2024/11/19-15:25:40.175675 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00228/att-00/compact.done) = No such file or directory: 7.205 sec on http probe failed: 3, retry 2024/11/19-15:25:40.382117 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-01 2024/11/19-15:25:40.382207 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00226/att-01/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:25:41.216218 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-01 2024/11/19-15:25:41.216317 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00223/att-01/compact.done) = No such file or directory: 11.200 sec on http probe failed: 5, giveup 2024/11/19-15:25:41.216337 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00223/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.826 sec, real 11.507 (submit 298.653 ms wait 11.200 s), input 126.826 MiB, send shutdown 2024/11/19-15:25:41.224099 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-01 2024/11/19-15:25:41.224313 139851148035840 [dcompact_etcd.cc:1773] job-00223/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:41.428326 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-01 2024/11/19-15:25:41.428393 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00227/att-01/compact.done) = No such file or directory: 7.184 sec on http probe failed: 3, retry 2024/11/19-15:25:41.501904 139851148035840 [dcompact_etcd.cc:1163] job-00223/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.278 sec, input = 126.826 MiB 2024/11/19-15:25:41.570259 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-01 2024/11/19-15:25:41.570350 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00224/att-01/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:25:41.570368 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00224/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.000 sec, real 11.479 (submit 262.380 ms wait 11.207 s), input 127.000 MiB, send shutdown 2024/11/19-15:25:41.577589 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-01 2024/11/19-15:25:41.577811 139851158521600 [dcompact_etcd.cc:1773] job-00224/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:41.645345 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-00 2024/11/19-15:25:41.645404 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00229/att-00/compact.done) = No such file or directory: 5.188 sec on http probe failed: 2, retry 2024/11/19-15:25:41.875878 139851158521600 [dcompact_etcd.cc:1163] job-00224/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.298 sec, input = 127.000 MiB 2024/11/19-15:25:42.034370 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-01 2024/11/19-15:25:42.034431 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00225/att-01/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:25:42.180901 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-00 2024/11/19-15:25:42.180964 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00228/att-00/compact.done) = No such file or directory: 9.214 sec on http probe failed: 4, retry 2024/11/19-15:25:42.387281 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-01 2024/11/19-15:25:42.387338 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00226/att-01/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:25:43.434617 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-01 2024/11/19-15:25:43.434718 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00227/att-01/compact.done) = No such file or directory: 9.192 sec on http probe failed: 4, retry 2024/11/19-15:25:43.654230 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-00 2024/11/19-15:25:43.654301 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00229/att-00/compact.done) = No such file or directory: 7.196 sec on http probe failed: 3, retry 2024/11/19-15:25:44.043838 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-01 2024/11/19-15:25:44.043929 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00225/att-01/compact.done) = No such file or directory: 11.209 sec on http probe failed: 5, giveup 2024/11/19-15:25:44.043948 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00225/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.866 sec, real 11.498 (submit 279.152 ms wait 11.209 s), input 126.866 MiB, send shutdown 2024/11/19-15:25:44.054964 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-01 2024/11/19-15:25:44.055186 139850978166528 [dcompact_etcd.cc:1773] job-00225/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:44.190548 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-00 2024/11/19-15:25:44.190630 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00228/att-00/compact.done) = No such file or directory: 11.220 sec on http probe failed: 5, giveup 2024/11/19-15:25:44.190649 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00228/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.892 sec, real 11.526 (submit 296.186 ms wait 11.220 s), input 126.892 MiB, send shutdown 2024/11/19-15:25:44.198612 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-00 2024/11/19-15:25:44.198825 139851169007360 [dcompact_etcd.cc:1773] job-00228/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:44.327005 139850978166528 [dcompact_etcd.cc:1163] job-00225/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.272 sec, input = 126.866 MiB 2024/11/19-15:25:44.395375 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-01 2024/11/19-15:25:44.395433 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00226/att-01/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:25:44.395450 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00226/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.998 sec, real 11.490 (submit 280.325 ms wait 11.202 s), input 126.998 MiB, send shutdown 2024/11/19-15:25:44.403898 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-01 2024/11/19-15:25:44.404106 139851179493120 [dcompact_etcd.cc:1773] job-00226/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:44.469362 139851169007360 [dcompact_etcd.cc:1163] job-00228/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.270 sec, input = 126.892 MiB 2024/11/19-15:25:44.672551 139851179493120 [dcompact_etcd.cc:1163] job-00226/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.268 sec, input = 126.998 MiB 2024/11/19-15:25:44.681557 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-02 2024/11/19-15:25:44.681621 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00223/att-02/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:25:45.058125 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-02 2024/11/19-15:25:45.058203 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00224/att-02/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:25:45.444931 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-01 2024/11/19-15:25:45.445030 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00227/att-01/compact.done) = No such file or directory: 11.199 sec on http probe failed: 5, giveup 2024/11/19-15:25:45.445048 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00227/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.878 sec, real 11.491 (submit 282.238 ms wait 11.199 s), input 126.878 MiB, send shutdown 2024/11/19-15:25:45.450129 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-01 2024/11/19-15:25:45.450338 139851127064320 [dcompact_etcd.cc:1773] job-00227/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:45.662350 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-00 2024/11/19-15:25:45.662417 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00229/att-00/compact.done) = No such file or directory: 9.205 sec on http probe failed: 4, retry 2024/11/19-15:25:45.746588 139851127064320 [dcompact_etcd.cc:1163] job-00227/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.296 sec, input = 126.878 MiB 2024/11/19-15:25:46.690512 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-02 2024/11/19-15:25:46.690610 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00223/att-02/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:25:47.063899 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-02 2024/11/19-15:25:47.063976 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00224/att-02/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:25:47.503914 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-02 2024/11/19-15:25:47.504017 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00225/att-02/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:25:47.651105 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-01 2024/11/19-15:25:47.651197 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00228/att-01/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:25:47.666687 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-00 2024/11/19-15:25:47.666748 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00229/att-00/compact.done) = No such file or directory: 11.213 sec on http probe failed: 5, giveup 2024/11/19-15:25:47.666766 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00229/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.980 sec, real 11.501 (submit 283.411 ms wait 11.213 s), input 126.980 MiB, send shutdown 2024/11/19-15:25:47.675056 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-00 2024/11/19-15:25:47.675265 139851137550080 [dcompact_etcd.cc:1773] job-00229/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:47.853044 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-02 2024/11/19-15:25:47.853143 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00226/att-02/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:25:47.931766 139851137550080 [dcompact_etcd.cc:1163] job-00229/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.256 sec, input = 126.980 MiB 2024/11/19-15:25:48.699686 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-02 2024/11/19-15:25:48.699764 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00223/att-02/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:25:48.924836 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-02 2024/11/19-15:25:48.924918 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00227/att-02/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:25:49.072002 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-02 2024/11/19-15:25:49.072107 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00224/att-02/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:25:49.512548 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-02 2024/11/19-15:25:49.512637 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00225/att-02/compact.done) = No such file or directory: 5.177 sec on http probe failed: 2, retry 2024/11/19-15:25:49.656299 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-01 2024/11/19-15:25:49.656366 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00228/att-01/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:25:49.861790 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-02 2024/11/19-15:25:49.861847 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00226/att-02/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:25:50.706230 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-02 2024/11/19-15:25:50.706311 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00223/att-02/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:25:50.930348 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-02 2024/11/19-15:25:50.930415 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00227/att-02/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:25:51.081595 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-02 2024/11/19-15:25:51.081656 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00224/att-02/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:25:51.111690 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-01 2024/11/19-15:25:51.111754 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00229/att-01/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:25:51.519640 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-02 2024/11/19-15:25:51.519722 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00225/att-02/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:25:51.665671 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-01 2024/11/19-15:25:51.665742 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00228/att-01/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:25:51.871438 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-02 2024/11/19-15:25:51.871518 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00226/att-02/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:25:52.712103 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-02 2024/11/19-15:25:52.712216 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00223/att-02/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:25:52.712236 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00223/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.826 sec, real 11.488 (submit 277.516 ms wait 11.204 s), input 126.826 MiB, send shutdown 2024/11/19-15:25:52.716881 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00223/att-02 2024/11/19-15:25:52.717122 139851148035840 [dcompact_etcd.cc:1773] job-00223/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:52.719025 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00223) = success 2024/11/19-15:25:52.719032 139851148035840 [dcompact_etcd.cc:1773] job-00223/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:25:52.719051 139851148035840 (Original Log Time 2024/11/19-15:25:18.199026) EVENT_LOG_v1 {"time_micros": 1732001118199018, "job": 223, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5036, 5039, 5041, 5043], "score": 5.2756, "input_data_size": 134098944, "oldest_snapshot_seqno": 146919500} 2024/11/19-15:25:52.719058 139851148035840 [compaction_job.cc:2437] [default] [JOB 223] Compacting 4@1 files to L2, score 5.28, subcompactions 1 : 1 2024/11/19-15:25:52.719065 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 221 Base level 1, inputs: [5036(31MB) 5039(31MB) 5041(31MB) 5043(31MB)] 2024/11/19-15:25:52.719275 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005207.sst) /storage/mytopling/..//tmp/Topling-82038-kC80Bl 2024/11/19-15:25:52.772207 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005207.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:52.775638 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005207.sst): index pass time = 0.00's, 152.838'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64714 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:25:52.857604 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005207.sst): first pass time = 0.14's, 485.186'MB/sec 2024/11/19-15:25:52.857659 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005207.sst): old prealloc_size = 73819750, real_size = 66151712 2024/11/19-15:25:52.857718 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005207.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64714 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:25:52.857764 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005207.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.086 sec, Key+Value = 0.067 GB 2024/11/19-15:25:52.917374 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005207.sst): second pass time = 0.00's,215080480.392'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1107.802'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2000.399'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.378'MB/sec write SST data time = 0.02's, 3935.808'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 215080480.392'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64714 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 64714 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0464 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 14.496106 GB avg = 0.251 KB (by entry num) total key len = 1.495162 GB avg = 0.026 KB total ukey len = 1.032682 GB avg = 0.018 KB total ukey num = 0.057809905 Billion total entry num = 0.057809905 Billion write speed all = 36.681494013 MB/sec (with seq num) write speed all = 35.620638216 MB/sec (without seq num) 2024/11/19-15:25:52.936184 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-02 2024/11/19-15:25:52.936251 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00227/att-02/compact.done) = No such file or directory: 7.184 sec on http probe failed: 3, retry 2024/11/19-15:25:53.039194 139851148035840 [compaction_job.cc:1948] [default] [JOB 223] Generated table #5207: 64714 keys, 66136964 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:53.039251 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001153039225, "cf_name": "default", "job": 223, "event": "table_file_creation", "file_number": 5207, "file_size": 66136964, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025168, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294280, "raw_average_key_size": 20, "raw_value_size": 65814627, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64714, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001152, "oldest_key_time": 0, "file_creation_time": 1732001152, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5207, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:53.053650 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005208.sst) /storage/mytopling/..//tmp/Topling-82038-xAkQw8 2024/11/19-15:25:53.088232 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-02 2024/11/19-15:25:53.088331 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00224/att-02/compact.done) = No such file or directory: 11.206 sec on http probe failed: 5, giveup 2024/11/19-15:25:53.088358 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00224/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.000 sec, real 11.510 (submit 297.986 ms wait 11.206 s), input 127.000 MiB, send shutdown 2024/11/19-15:25:53.096728 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00224/att-02 2024/11/19-15:25:53.096998 139851158521600 [dcompact_etcd.cc:1773] job-00224/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:53.098890 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00224) = success 2024/11/19-15:25:53.098906 139851158521600 [dcompact_etcd.cc:1773] job-00224/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:25:53.098938 139851158521600 (Original Log Time 2024/11/19-15:25:18.575353) EVENT_LOG_v1 {"time_micros": 1732001118575344, "job": 224, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5045, 5050, 5051, 5052], "score": 5.02583, "input_data_size": 134099008, "oldest_snapshot_seqno": 146919500} 2024/11/19-15:25:53.098950 139851158521600 [compaction_job.cc:2437] [default] [JOB 224] Compacting 4@1 files to L2, score 5.03, subcompactions 1 : 1 2024/11/19-15:25:53.098963 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 222 Base level 1, inputs: [5045(31MB) 5050(31MB) 5051(31MB) 5052(31MB)] 2024/11/19-15:25:53.099250 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005209.sst) /storage/mytopling/..//tmp/Topling-82038-7i5s46 2024/11/19-15:25:53.105633 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005208.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:25:53.105720 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005208.sst): index pass time = 0.00's,7793.761'MB/sec index type = UintIndex_AllOne usrkeys = 63525 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:25:53.120519 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-01 2024/11/19-15:25:53.120596 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00229/att-01/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:25:53.168224 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005209.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:53.171841 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005208.sst): first pass time = 0.12's, 557.479'MB/sec 2024/11/19-15:25:53.171888 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005208.sst): old prealloc_size = 73819750, real_size = 64830584 2024/11/19-15:25:53.171911 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005208.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63525 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0646 GB zip-val = 0.0648 GB avg-val =1017.04 avg-zval =1020.30 2024/11/19-15:25:53.173642 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005209.sst): index pass time = 0.01's, 96.208'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64715 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:25:53.226436 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005208.sst): second pass time = 0.00's,146502639.456'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1188.598'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 6.594'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.405'MB/sec write SST data time = 0.02's, 4251.034'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 146502639.456'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63525 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.04 avg-zval = 1020.30 usrkeys = 63525 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.04 avg-zval = 1020.30 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0000 GB value = 0.0648 GB dict = 0.00 MB all = 0.0648 GB } UnZip/Zip{ index =3970.3125 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 14.560714 GB avg = 0.252 KB (by entry num) total key len = 1.496432 GB avg = 0.026 KB total ukey len = 1.033445 GB avg = 0.018 KB total ukey num = 0.057873430 Billion total entry num = 0.057873430 Billion write speed all = 36.806514743 MB/sec (with seq num) write speed all = 35.745245588 MB/sec (without seq num) 2024/11/19-15:25:53.238615 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005209.sst): first pass time = 0.14's, 481.590'MB/sec 2024/11/19-15:25:53.238662 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005209.sst): old prealloc_size = 73819750, real_size = 66152160 2024/11/19-15:25:53.238720 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005209.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64715 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.00 avg-zval =1020.25 2024/11/19-15:25:53.238759 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005209.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.071 sec, Key+Value = 0.067 GB 2024/11/19-15:25:53.241596 139851148035840 [compaction_job.cc:1948] [default] [JOB 223] Generated table #5208: 63525 keys, 64815626 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:53.241648 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001153241625, "cf_name": "default", "job": 223, "event": "table_file_creation", "file_number": 5208, "file_size": 64815626, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64814336, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1270500, "raw_average_key_size": 20, "raw_value_size": 64607664, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63525, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001153, "oldest_key_time": 0, "file_creation_time": 1732001153, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5208, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:53.255912 139851148035840 [compaction_job.cc:703] job-00223: subcompact[0], size: 0.130953 G, files: /storage/mytopling/.rocksdb [005207,005208] 2024/11/19-15:25:53.256674 139851148035840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 10151255 2024/11/19-15:25:53.256830 139851148035840 (Original Log Time 2024/11/19-15:25:52.719092) EVENT_LOG_v1 {"time_micros": 1732001152719078, "job": 223, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5036, 5039, 5041, 5043], "score": 5.2756, "input_data_size": 134098944, "oldest_snapshot_seqno": 146919500} 2024/11/19-15:25:53.256840 139851148035840 (Original Log Time 2024/11/19-15:25:53.256305) [compaction_job.cc:2022] [default] [JOB 223] Compacted 4@1 files to L2 => 130952590 bytes 2024/11/19-15:25:53.256846 139851148035840 (Original Log Time 2024/11/19-15:25:53.256640) [compaction_job.cc:1224] [default] compacted to: files[5 81 81 55 0 0 0] max score 3.53 (143 files need compaction), MB/sec: 249.8 rd, 243.9 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(124.9 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128239, records dropped: 0 output_compression: Snappy 2024/11/19-15:25:53.256871 139851148035840 (Original Log Time 2024/11/19-15:25:53.256666) EVENT_LOG_v1 {"time_micros": 1732001153256653, "cf": "default", "job": 223, "event": "compaction_finished", "compaction_time_micros": 536842, "compaction_time_cpu_micros": 402097, "output_level": 2, "num_output_files": 2, "total_output_size": 130952590, "num_input_records": 128239, "num_output_records": 128239, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 81, 81, 55, 0, 0, 0]} 2024/11/19-15:25:53.282126 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005133.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:53.282158 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001153282154, "job": 223, "event": "table_file_deletion", "file_number": 5133} 2024/11/19-15:25:53.301931 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005137.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:53.301959 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001153301956, "job": 223, "event": "table_file_deletion", "file_number": 5137} 2024/11/19-15:25:53.305858 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005142.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:53.305881 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001153305878, "job": 223, "event": "table_file_deletion", "file_number": 5142} 2024/11/19-15:25:53.306035 139851148035840 [compaction_job.cc:2437] [default] [JOB 230] Compacting 4@1 files to L2, score 3.53, subcompactions 1 : 1 2024/11/19-15:25:53.306048 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 228 Base level 1, inputs: [5081(31MB) 5082(31MB) 5083(31MB) 5085(31MB)] 2024/11/19-15:25:53.307598 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005209.sst): second pass time = 0.00's,291217163.717'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 959.587'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2042.213'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.313'MB/sec write SST data time = 0.02's, 3854.467'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 291217163.717'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64715 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.25 usrkeys = 64715 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0465 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 14.626529 GB avg = 0.252 KB (by entry num) total key len = 1.497726 GB avg = 0.026 KB total ukey len = 1.034221 GB avg = 0.018 KB total ukey num = 0.057938145 Billion total entry num = 0.057938145 Billion write speed all = 36.953469051 MB/sec (with seq num) write speed all = 35.891210801 MB/sec (without seq num) 2024/11/19-15:25:53.311116 139851158521600 [compaction_job.cc:1948] [default] [JOB 224] Generated table #5209: 64715 keys, 66137412 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:53.311158 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001153311137, "cf_name": "default", "job": 224, "event": "table_file_creation", "file_number": 5209, "file_size": 66137412, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025616, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294300, "raw_average_key_size": 20, "raw_value_size": 65815079, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64715, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001153, "oldest_key_time": 0, "file_creation_time": 1732001153, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5209, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:53.325312 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005210.sst) /storage/mytopling/..//tmp/Topling-82038-Cp5RxC 2024/11/19-15:25:53.377007 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005210.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:25:53.377098 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005210.sst): index pass time = 0.00's,7993.977'MB/sec index type = UintIndex_AllOne usrkeys = 63705 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:25:53.442246 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005210.sst): first pass time = 0.12's, 565.010'MB/sec 2024/11/19-15:25:53.442299 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005210.sst): old prealloc_size = 73819750, real_size = 65009544 2024/11/19-15:25:53.442321 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005210.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63705 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0648 GB zip-val = 0.0650 GB avg-val =1016.97 avg-zval =1020.22 2024/11/19-15:25:53.496879 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005210.sst): second pass time = 0.00's,108701421.141'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1191.633'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.589'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.409'MB/sec write SST data time = 0.02's, 4176.115'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 108701421.141'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63705 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.97 avg-zval = 1020.22 usrkeys = 63705 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.97 avg-zval = 1020.22 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0648 GB dict = 0.00 MB all = 0.0661 GB } __ZipSize{ index = 0.0000 GB value = 0.0650 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index =3981.5625 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 14.691315 GB avg = 0.253 KB (by entry num) total key len = 1.499001 GB avg = 0.026 KB total ukey len = 1.034986 GB avg = 0.018 KB total ukey num = 0.058001850 Billion total entry num = 0.058001850 Billion write speed all = 37.088778510 MB/sec (with seq num) write speed all = 36.025813317 MB/sec (without seq num) 2024/11/19-15:25:53.511563 139851158521600 [compaction_job.cc:1948] [default] [JOB 224] Generated table #5210: 63705 keys, 64994634 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:53.511617 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001153511593, "cf_name": "default", "job": 224, "event": "table_file_creation", "file_number": 5210, "file_size": 64994634, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64993296, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1274100, "raw_average_key_size": 20, "raw_value_size": 64786047, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63705, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001153, "oldest_key_time": 0, "file_creation_time": 1732001153, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5210, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:53.526443 139851158521600 [compaction_job.cc:703] job-00224: subcompact[0], size: 0.131132 G, files: /storage/mytopling/.rocksdb [005209,005210] 2024/11/19-15:25:53.529247 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-02 2024/11/19-15:25:53.529326 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00225/att-02/compact.done) = No such file or directory: 9.193 sec on http probe failed: 4, retry 2024/11/19-15:25:53.535656 139851158521600 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 10465211 2024/11/19-15:25:53.535817 139851158521600 (Original Log Time 2024/11/19-15:25:53.099003) EVENT_LOG_v1 {"time_micros": 1732001153098980, "job": 224, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5045, 5050, 5051, 5052], "score": 5.02583, "input_data_size": 134099008, "oldest_snapshot_seqno": 146919500} 2024/11/19-15:25:53.535826 139851158521600 (Original Log Time 2024/11/19-15:25:53.535222) [compaction_job.cc:2022] [default] [JOB 224] Compacted 4@1 files to L2 => 131132046 bytes 2024/11/19-15:25:53.535831 139851158521600 (Original Log Time 2024/11/19-15:25:53.535616) [compaction_job.cc:1224] [default] compacted to: files[5 77 83 55 0 0 0] max score 3.28 (141 files need compaction), MB/sec: 313.7 rd, 306.8 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.1 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128420, records dropped: 0 output_compression: Snappy 2024/11/19-15:25:53.535856 139851158521600 (Original Log Time 2024/11/19-15:25:53.535648) EVENT_LOG_v1 {"time_micros": 1732001153535630, "cf": "default", "job": 224, "event": "compaction_finished", "compaction_time_micros": 427454, "compaction_time_cpu_micros": 410870, "output_level": 2, "num_output_files": 2, "total_output_size": 131132046, "num_input_records": 128420, "num_output_records": 128420, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 77, 83, 55, 0, 0, 0]} 2024/11/19-15:25:53.563781 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005102.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:53.563807 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001153563804, "job": 224, "event": "table_file_deletion", "file_number": 5102} 2024/11/19-15:25:53.579887 139851148035840 [dcompact_etcd.cc:1163] job-00230/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.273 sec, input = 126.896 MiB 2024/11/19-15:25:53.596465 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005104.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:53.596493 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001153596490, "job": 224, "event": "table_file_deletion", "file_number": 5104} 2024/11/19-15:25:53.600503 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005106.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:53.600523 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001153600521, "job": 224, "event": "table_file_deletion", "file_number": 5106} 2024/11/19-15:25:53.604728 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005108.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:53.604747 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001153604745, "job": 224, "event": "table_file_deletion", "file_number": 5108} 2024/11/19-15:25:53.604874 139851158521600 [compaction_job.cc:2437] [default] [JOB 231] Compacting 4@1 files to L2, score 3.28, subcompactions 1 : 1 2024/11/19-15:25:53.604882 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 229 Base level 1, inputs: [5086(31MB) 5087(31MB) 5089(31MB) 5090(31MB)] 2024/11/19-15:25:53.670474 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-01 2024/11/19-15:25:53.670537 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00228/att-01/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:25:53.880220 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-02 2024/11/19-15:25:53.880325 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00226/att-02/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:25:53.937131 139851158521600 [dcompact_etcd.cc:1163] job-00231/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.332 sec, input = 126.998 MiB 2024/11/19-15:25:54.945706 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-02 2024/11/19-15:25:54.945789 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00227/att-02/compact.done) = No such file or directory: 9.190 sec on http probe failed: 4, retry 2024/11/19-15:25:55.130495 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-01 2024/11/19-15:25:55.130568 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00229/att-01/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:25:55.534338 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-02 2024/11/19-15:25:55.534433 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00225/att-02/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:25:55.534458 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00225/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.866 sec, real 11.479 (submit 271.747 ms wait 11.202 s), input 126.866 MiB, send shutdown 2024/11/19-15:25:55.543146 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00225/att-02 2024/11/19-15:25:55.543362 139850978166528 [dcompact_etcd.cc:1773] job-00225/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:55.545226 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00225) = success 2024/11/19-15:25:55.545233 139850978166528 [dcompact_etcd.cc:1773] job-00225/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:25:55.545251 139850978166528 (Original Log Time 2024/11/19-15:25:21.011552) EVENT_LOG_v1 {"time_micros": 1732001121011544, "job": 225, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5053, 5055, 5056, 5057], "score": 4.77605, "input_data_size": 134100224, "oldest_snapshot_seqno": 146981429} 2024/11/19-15:25:55.545258 139850978166528 [compaction_job.cc:2437] [default] [JOB 225] Compacting 4@1 files to L2, score 4.78, subcompactions 1 : 1 2024/11/19-15:25:55.545266 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 223 Base level 1, inputs: [5053(31MB) 5055(31MB) 5056(31MB) 5057(31MB)] 2024/11/19-15:25:55.545510 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005211.sst) /storage/mytopling/..//tmp/Topling-82038-ABHc8f 2024/11/19-15:25:55.598782 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005211.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:55.602174 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005211.sst): index pass time = 0.00's, 154.117'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64711 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:25:55.665204 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005211.sst): first pass time = 0.12's, 560.736'MB/sec 2024/11/19-15:25:55.665254 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005211.sst): old prealloc_size = 73819750, real_size = 66152288 2024/11/19-15:25:55.665313 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005211.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64711 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.06 avg-zval =1020.32 2024/11/19-15:25:55.665383 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005211.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.067 GB 2024/11/19-15:25:55.680057 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-01 2024/11/19-15:25:55.680137 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00228/att-01/compact.done) = No such file or directory: 11.201 sec on http probe failed: 5, giveup 2024/11/19-15:25:55.680156 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00228/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.892 sec, real 11.481 (submit 270.463 ms wait 11.201 s), input 126.892 MiB, send shutdown 2024/11/19-15:25:55.684743 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-01 2024/11/19-15:25:55.684965 139851169007360 [dcompact_etcd.cc:1773] job-00228/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:55.732850 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005211.sst): second pass time = 0.00's,223861261.905'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 978.866'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2026.926'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.366'MB/sec write SST data time = 0.02's, 2835.471'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 223861261.905'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64711 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.06 avg-zval = 1020.32 usrkeys = 64711 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.06 avg-zval = 1020.32 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0461 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 14.757130 GB avg = 0.254 KB (by entry num) total key len = 1.500295 GB avg = 0.026 KB total ukey len = 1.035762 GB avg = 0.018 KB total ukey num = 0.058066561 Billion total entry num = 0.058066561 Billion write speed all = 37.052722155 MB/sec (with seq num) write speed all = 35.993994037 MB/sec (without seq num) 2024/11/19-15:25:55.737810 139850978166528 [compaction_job.cc:1948] [default] [JOB 225] Generated table #5211: 64711 keys, 66137540 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:55.737876 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001155737841, "cf_name": "default", "job": 225, "event": "table_file_creation", "file_number": 5211, "file_size": 66137540, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025744, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294220, "raw_average_key_size": 20, "raw_value_size": 65815211, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64711, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001155, "oldest_key_time": 0, "file_creation_time": 1732001155, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5211, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:55.754480 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005212.sst) /storage/mytopling/..//tmp/Topling-82038-rTmU3q 2024/11/19-15:25:55.806430 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005212.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:25:55.809696 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005212.sst): index pass time = 0.00's, 157.359'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63570 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:25:55.873385 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005212.sst): first pass time = 0.12's, 554.521'MB/sec 2024/11/19-15:25:55.873446 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005212.sst): old prealloc_size = 73819750, real_size = 64979168 2024/11/19-15:25:55.873517 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005212.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63570 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0646 GB zip-val = 0.0649 GB avg-val =1016.96 avg-zval =1020.21 2024/11/19-15:25:55.873595 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005212.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.066 GB 2024/11/19-15:25:55.884961 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-02 2024/11/19-15:25:55.885090 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00226/att-02/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:25:55.885117 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00226/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.998 sec, real 11.481 (submit 268.369 ms wait 11.208 s), input 126.998 MiB, send shutdown 2024/11/19-15:25:55.893635 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00226/att-02 2024/11/19-15:25:55.933392 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005212.sst): second pass time = 0.00's,131934655.102'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1083.957'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1635.227'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.372'MB/sec write SST data time = 0.02's, 3785.165'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 131934655.102'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63570 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.96 avg-zval = 1020.21 usrkeys = 63570 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.96 avg-zval = 1020.21 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7921 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0445 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 14.821778 GB avg = 0.255 KB (by entry num) total key len = 1.501566 GB avg = 0.026 KB total ukey len = 1.036525 GB avg = 0.018 KB total ukey num = 0.058130131 Billion total entry num = 0.058130131 Billion write speed all = 37.185963344 MB/sec (with seq num) write speed all = 36.126560397 MB/sec (without seq num) 2024/11/19-15:25:55.934852 139851179493120 [dcompact_etcd.cc:1773] job-00226/att-02: CleanFiles 0.041 sec : 0.000 2024/11/19-15:25:55.942915 139851169007360 [dcompact_etcd.cc:1163] job-00228/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.258 sec, input = 126.892 MiB 2024/11/19-15:25:55.953003 139850978166528 [compaction_job.cc:1948] [default] [JOB 225] Generated table #5212: 63570 keys, 64964676 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:55.953086 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001155953049, "cf_name": "default", "job": 225, "event": "table_file_creation", "file_number": 5212, "file_size": 64964676, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64854800, "index_size": 108288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1271400, "raw_average_key_size": 20, "raw_value_size": 64647981, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63570, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001155, "oldest_key_time": 0, "file_creation_time": 1732001155, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5212, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:55.964737 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00226) = success 2024/11/19-15:25:55.964764 139851179493120 [dcompact_etcd.cc:1773] job-00226/att-03: CleanFiles 0.030 sec : 0.000 2024/11/19-15:25:55.964808 139851179493120 (Original Log Time 2024/11/19-15:25:21.391612) EVENT_LOG_v1 {"time_micros": 1732001121391605, "job": 226, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5060, 5061, 5062, 5064], "score": 4.52627, "input_data_size": 134098560, "oldest_snapshot_seqno": 147017398} 2024/11/19-15:25:55.964825 139851179493120 [compaction_job.cc:2437] [default] [JOB 226] Compacting 4@1 files to L2, score 4.53, subcompactions 1 : 1 2024/11/19-15:25:55.964846 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 224 Base level 1, inputs: [5060(31MB) 5061(31MB) 5062(31MB) 5064(31MB)] 2024/11/19-15:25:55.965371 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005213.sst) /storage/mytopling/..//tmp/Topling-82038-jMo4O7 2024/11/19-15:25:55.974587 139850978166528 [compaction_job.cc:703] job-00225: subcompact[0], size: 0.131102 G, files: /storage/mytopling/.rocksdb [005211,005212] 2024/11/19-15:25:55.978446 139850978166528 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 10788877 2024/11/19-15:25:55.978720 139850978166528 (Original Log Time 2024/11/19-15:25:55.545291) EVENT_LOG_v1 {"time_micros": 1732001155545277, "job": 225, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5053, 5055, 5056, 5057], "score": 4.77605, "input_data_size": 134100224, "oldest_snapshot_seqno": 146981429} 2024/11/19-15:25:55.978740 139850978166528 (Original Log Time 2024/11/19-15:25:55.977945) [compaction_job.cc:2022] [default] [JOB 225] Compacted 4@1 files to L2 => 131102216 bytes 2024/11/19-15:25:55.978750 139850978166528 (Original Log Time 2024/11/19-15:25:55.978399) [compaction_job.cc:1224] [default] compacted to: files[5 73 85 55 0 0 0] max score 3.14 (139 files need compaction), MB/sec: 312.4 rd, 305.4 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128281, records dropped: 0 output_compression: Snappy 2024/11/19-15:25:55.978794 139850978166528 (Original Log Time 2024/11/19-15:25:55.978436) EVENT_LOG_v1 {"time_micros": 1732001155978418, "cf": "default", "job": 225, "event": "compaction_finished", "compaction_time_micros": 429316, "compaction_time_cpu_micros": 407870, "output_level": 2, "num_output_files": 2, "total_output_size": 131102216, "num_input_records": 128281, "num_output_records": 128281, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 73, 85, 55, 0, 0, 0]} 2024/11/19-15:25:55.986395 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005113.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:55.986442 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001155986436, "job": 225, "event": "table_file_deletion", "file_number": 5113} 2024/11/19-15:25:56.018524 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005118.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:56.018558 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001156018555, "job": 225, "event": "table_file_deletion", "file_number": 5118} 2024/11/19-15:25:56.024850 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005123.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:56.024880 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001156024876, "job": 225, "event": "table_file_deletion", "file_number": 5123} 2024/11/19-15:25:56.031052 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005128.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:56.031080 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001156031077, "job": 225, "event": "table_file_deletion", "file_number": 5128} 2024/11/19-15:25:56.031261 139850978166528 [compaction_job.cc:2437] [default] [JOB 232] Compacting 1@2 files to L3, score 3.14, subcompactions 1 : 1 2024/11/19-15:25:56.031274 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 230 Base level 2, inputs: [5169(3971B)] 2024/11/19-15:25:56.039952 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005213.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:56.040064 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005213.sst): index pass time = 0.00's,5620.854'MB/sec index type = UintIndex_AllOne usrkeys = 64715 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:25:56.106036 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005213.sst): first pass time = 0.14's, 477.182'MB/sec 2024/11/19-15:25:56.106088 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005213.sst): old prealloc_size = 73819750, real_size = 66042584 2024/11/19-15:25:56.106111 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005213.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64715 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:25:56.160629 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005213.sst): second pass time = 0.00's,102356992.224'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1210.872'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.746'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.416'MB/sec write SST data time = 0.02's, 4230.195'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 102356992.224'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64715 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 64715 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.6875 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 14.887594 GB avg = 0.256 KB (by entry num) total key len = 1.502860 GB avg = 0.026 KB total ukey len = 1.037302 GB avg = 0.018 KB total ukey num = 0.058194846 Billion total entry num = 0.058194846 Billion write speed all = 37.319525994 MB/sec (with seq num) write speed all = 36.259492387 MB/sec (without seq num) 2024/11/19-15:25:56.184036 139851179493120 [compaction_job.cc:1948] [default] [JOB 226] Generated table #5213: 64715 keys, 66027402 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:56.184088 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001156184064, "cf_name": "default", "job": 226, "event": "table_file_creation", "file_number": 5213, "file_size": 66027402, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66026080, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294300, "raw_average_key_size": 20, "raw_value_size": 65815546, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64715, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001155, "oldest_key_time": 0, "file_creation_time": 1732001155, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5213, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:56.200075 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005214.sst) /storage/mytopling/..//tmp/Topling-82038-BJ7BAz 2024/11/19-15:25:56.255358 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005214.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:25:56.258689 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005214.sst): index pass time = 0.00's, 154.478'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63701 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:25:56.320076 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005214.sst): first pass time = 0.12's, 550.562'MB/sec 2024/11/19-15:25:56.320124 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005214.sst): old prealloc_size = 73819750, real_size = 65115056 2024/11/19-15:25:56.320178 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005214.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63701 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0648 GB zip-val = 0.0650 GB avg-val =1016.99 avg-zval =1020.24 2024/11/19-15:25:56.320244 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005214.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.065 sec, Key+Value = 0.066 GB 2024/11/19-15:25:56.364152 139850978166528 [dcompact_etcd.cc:1163] job-00232/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.332 sec, input = 2.559 KiB 2024/11/19-15:25:56.373428 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-00 2024/11/19-15:25:56.373489 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00232/att-00/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:25:56.378300 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005214.sst): second pass time = 0.00's,233874592.058'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1119.246'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2085.071'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.383'MB/sec write SST data time = 0.02's, 3939.020'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 233874592.058'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63701 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 usrkeys = 63701 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7938 UnZipSize{ index = 0.0008 GB value = 0.0648 GB dict = 0.00 MB all = 0.0661 GB } __ZipSize{ index = 0.0001 GB value = 0.0650 GB dict = 0.00 MB all = 0.0651 GB } UnZip/Zip{ index = 7.0466 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 14.952377 GB avg = 0.257 KB (by entry num) total key len = 1.504134 GB avg = 0.026 KB total ukey len = 1.038066 GB avg = 0.018 KB total ukey num = 0.058258547 Billion total entry num = 0.058258547 Billion write speed all = 37.451370477 MB/sec (with seq num) write speed all = 36.390702232 MB/sec (without seq num) 2024/11/19-15:25:56.381565 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-00 2024/11/19-15:25:56.381613 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00232/att-00/compact.done) = No such file or directory: 0.009 sec on http probe failed: 2, retry 2024/11/19-15:25:56.381862 139851179493120 [compaction_job.cc:1948] [default] [JOB 226] Generated table #5214: 63701 keys, 65100548 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:56.381910 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001156381887, "cf_name": "default", "job": 226, "event": "table_file_creation", "file_number": 5214, "file_size": 65100548, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64990496, "index_size": 108480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1274020, "raw_average_key_size": 20, "raw_value_size": 64783262, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63701, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001156, "oldest_key_time": 0, "file_creation_time": 1732001156, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5214, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:56.386118 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-00 2024/11/19-15:25:56.386165 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00232/att-00/compact.done) = No such file or directory: 0.018 sec on http probe failed: 3, retry 2024/11/19-15:25:56.394393 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-00 2024/11/19-15:25:56.394435 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00232/att-00/compact.done) = No such file or directory: 0.022 sec on http probe failed: 4, retry 2024/11/19-15:25:56.395996 139851179493120 [compaction_job.cc:703] job-00226: subcompact[0], size: 0.131128 G, files: /storage/mytopling/.rocksdb [005213,005214] 2024/11/19-15:25:56.396687 139851179493120 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 11122553 2024/11/19-15:25:56.396817 139851179493120 (Original Log Time 2024/11/19-15:25:55.964911) EVENT_LOG_v1 {"time_micros": 1732001155964869, "job": 226, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5060, 5061, 5062, 5064], "score": 4.52627, "input_data_size": 134098560, "oldest_snapshot_seqno": 147017398} 2024/11/19-15:25:56.396823 139851179493120 (Original Log Time 2024/11/19-15:25:56.396353) [compaction_job.cc:2022] [default] [JOB 226] Compacted 4@1 files to L2 => 131127950 bytes 2024/11/19-15:25:56.396827 139851179493120 (Original Log Time 2024/11/19-15:25:56.396660) [compaction_job.cc:1224] [default] compacted to: files[5 69 87 55 0 0 0] max score 3.22 (140 files need compaction), MB/sec: 311.1 rd, 304.2 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.1 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128416, records dropped: 0 output_compression: Snappy 2024/11/19-15:25:56.396843 139851179493120 (Original Log Time 2024/11/19-15:25:56.396681) EVENT_LOG_v1 {"time_micros": 1732001156396670, "cf": "default", "job": 226, "event": "compaction_finished", "compaction_time_micros": 431096, "compaction_time_cpu_micros": 406368, "output_level": 2, "num_output_files": 2, "total_output_size": 131127950, "num_input_records": 128416, "num_output_records": 128416, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 69, 87, 55, 0, 0, 0]} 2024/11/19-15:25:56.399570 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-00 2024/11/19-15:25:56.399618 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00232/att-00/compact.done) = No such file or directory: 0.030 sec on http probe failed: 5, giveup 2024/11/19-15:25:56.399631 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00232/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.368 (submit 332.297 ms wait 0.030 s), input 2.559 KiB, send shutdown 2024/11/19-15:25:56.401075 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005017.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:56.401098 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001156401095, "job": 226, "event": "table_file_deletion", "file_number": 5017} 2024/11/19-15:25:56.405151 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005146.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:56.405172 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001156405170, "job": 226, "event": "table_file_deletion", "file_number": 5146} 2024/11/19-15:25:56.407177 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-00 2024/11/19-15:25:56.407353 139850978166528 [dcompact_etcd.cc:1773] job-00232/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:56.409195 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005148.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:56.409214 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001156409212, "job": 226, "event": "table_file_deletion", "file_number": 5148} 2024/11/19-15:25:56.409725 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005150.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:56.409737 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001156409736, "job": 226, "event": "table_file_deletion", "file_number": 5150} 2024/11/19-15:25:56.409857 139851179493120 [compaction_job.cc:2437] [default] [JOB 233] Compacting 4@2 files to L3, score 3.22, subcompactions 1 : 1 2024/11/19-15:25:56.409865 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 231 Base level 2, inputs: [5174(62MB) 5175(61MB) 5183(62MB) 5184(61MB)] 2024/11/19-15:25:56.693413 139851179493120 [dcompact_etcd.cc:1163] job-00233/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.283 sec, input = 254.473 MiB 2024/11/19-15:25:56.716549 139850978166528 [dcompact_etcd.cc:1163] job-00232/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.309 sec, input = 2.559 KiB 2024/11/19-15:25:56.725003 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-01 2024/11/19-15:25:56.725064 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00232/att-01/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:25:56.733100 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-01 2024/11/19-15:25:56.733145 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00232/att-01/compact.done) = No such file or directory: 0.009 sec on http probe failed: 2, retry 2024/11/19-15:25:56.737698 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-01 2024/11/19-15:25:56.737737 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00232/att-01/compact.done) = No such file or directory: 0.017 sec on http probe failed: 3, retry 2024/11/19-15:25:56.745908 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-01 2024/11/19-15:25:56.745956 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00232/att-01/compact.done) = No such file or directory: 0.021 sec on http probe failed: 4, retry 2024/11/19-15:25:56.751058 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-01 2024/11/19-15:25:56.751102 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00232/att-01/compact.done) = No such file or directory: 0.030 sec on http probe failed: 5, giveup 2024/11/19-15:25:56.751115 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00232/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.344 (submit 309.125 ms wait 0.030 s), input 2.559 KiB, send shutdown 2024/11/19-15:25:56.758656 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-01 2024/11/19-15:25:56.758964 139850978166528 [dcompact_etcd.cc:1773] job-00232/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:56.760637 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-00 2024/11/19-15:25:56.760697 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00230/att-00/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:25:56.954725 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-02 2024/11/19-15:25:56.954819 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00227/att-02/compact.done) = No such file or directory: 11.199 sec on http probe failed: 5, giveup 2024/11/19-15:25:56.954838 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00227/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.878 sec, real 11.504 (submit 296.162 ms wait 11.199 s), input 126.878 MiB, send shutdown 2024/11/19-15:25:56.963019 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00227/att-02 2024/11/19-15:25:56.963238 139851127064320 [dcompact_etcd.cc:1773] job-00227/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:56.965136 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00227) = success 2024/11/19-15:25:56.965144 139851127064320 [dcompact_etcd.cc:1773] job-00227/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:25:56.965162 139851127064320 (Original Log Time 2024/11/19-15:25:22.412183) EVENT_LOG_v1 {"time_micros": 1732001122412175, "job": 227, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5065, 5066, 5068, 5069], "score": 4.27649, "input_data_size": 134099264, "oldest_snapshot_seqno": -1} 2024/11/19-15:25:56.965169 139851127064320 [compaction_job.cc:2437] [default] [JOB 227] Compacting 4@1 files to L2, score 4.28, subcompactions 1 : 1 2024/11/19-15:25:56.965177 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 225 Base level 1, inputs: [5065(31MB) 5066(31MB) 5068(31MB) 5069(31MB)] 2024/11/19-15:25:56.965423 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005215.sst) /storage/mytopling/..//tmp/Topling-82038-qtbzI1 2024/11/19-15:25:57.028416 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005215.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:25:57.031764 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005215.sst): index pass time = 0.00's, 156.324'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64720 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:25:57.049777 139850978166528 [dcompact_etcd.cc:1163] job-00232/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.291 sec, input = 2.559 KiB 2024/11/19-15:25:57.065310 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-02 2024/11/19-15:25:57.065390 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00232/att-02/compact.done) = No such file or directory: 0.009 sec on http probe failed: 1, retry 2024/11/19-15:25:57.074070 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-02 2024/11/19-15:25:57.074149 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00232/att-02/compact.done) = No such file or directory: 0.016 sec on http probe failed: 2, retry 2024/11/19-15:25:57.082018 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-02 2024/11/19-15:25:57.082085 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00232/att-02/compact.done) = No such file or directory: 0.024 sec on http probe failed: 3, retry 2024/11/19-15:25:57.089507 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-02 2024/11/19-15:25:57.089563 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00232/att-02/compact.done) = No such file or directory: 0.032 sec on http probe failed: 4, retry 2024/11/19-15:25:57.098832 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-02 2024/11/19-15:25:57.098893 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00232/att-02/compact.done) = No such file or directory: 0.040 sec on http probe failed: 5, giveup 2024/11/19-15:25:57.098910 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00232/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.340 (submit 290.744 ms wait 0.040 s), input 2.559 KiB, send shutdown 2024/11/19-15:25:57.102997 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005215.sst): first pass time = 0.14's, 487.851'MB/sec 2024/11/19-15:25:57.103044 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005215.sst): old prealloc_size = 73819750, real_size = 66152160 2024/11/19-15:25:57.103120 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005215.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64720 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.92 avg-zval =1020.17 2024/11/19-15:25:57.103197 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005215.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.075 sec, Key+Value = 0.067 GB 2024/11/19-15:25:57.107344 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00232/att-02 2024/11/19-15:25:57.107565 139850978166528 [dcompact_etcd.cc:1773] job-00232/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:25:57.109303 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00232) = success 2024/11/19-15:25:57.109315 139850978166528 [dcompact_etcd.cc:1773] job-00232/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:25:57.109333 139850978166528 (Original Log Time 2024/11/19-15:25:56.031293) EVENT_LOG_v1 {"time_micros": 1732001156031283, "job": 232, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [5169], "score": 3.13857, "input_data_size": 3971, "oldest_snapshot_seqno": 147986157} 2024/11/19-15:25:57.109341 139850978166528 [compaction_job.cc:2437] [default] [JOB 232] Compacting 1@2 files to L3, score 3.14, subcompactions 1 : 1 2024/11/19-15:25:57.109348 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 230 Base level 2, inputs: [5169(3971B)] 2024/11/19-15:25:57.109618 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005216.sst) /storage/mytopling/..//tmp/Topling-82038-3Ve5w5 2024/11/19-15:25:57.109738 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005216.sst): first pass time = 0.00's, 23.810'MB/sec 2024/11/19-15:25:57.109750 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005216.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:25:57.109768 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005216.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, dictZip workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:25:57.109829 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005216.sst): index pass time = 0.00's, 1.265'MB/sec index type = UintIndex_AllOne usrkeys = 20 min-keylen = 4 max-keylen = 4 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 8.00 2024/11/19-15:25:57.119785 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-00 2024/11/19-15:25:57.119859 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00231/att-00/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:25:57.130252 139850978166528 [top_zip_table_builder.cc:2752] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005216.sst): part-0: reject ZipStore ratio = 0.870 2024/11/19-15:25:57.130503 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005216.sst): old prealloc_size = 4368, real_size = 2944 2024/11/19-15:25:57.130522 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005216.sst): index type = UintIndex_AllOne , store type = ZipOffsetBlobStore usrkeys = 20 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 8.00 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 115.00 avg-zval = 128.80 2024/11/19-15:25:57.130685 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005216.sst): second pass time = 0.02's, 0.112'MB/sec, value only(87.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 22.238'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.110'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.422'MB/sec write SST data time = 0.00's, 26.798'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = 0.000'MB/sec zip my value time = 0.02's, unzip length = 0.000'GB zip my value throughput = 0.112'MB/sec zip pipeline throughput = 0.112'MB/sec entries = 20 avg-key = 16.00 avg-zkey = 9.60 avg-val = 115.00 avg-zval = 128.80 usrkeys = 20 avg-key = 8.00 avg-zkey = 9.60 avg-val = 115.00 avg-zval = 128.80 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 4 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.8333 value = 0.8929 dict = 0.00 all = 0.6720 } Zip/UnZip{ index = 1.2000 value = 1.1200 dict = 0.00 all = 1.4882 } ---------------------------- total value len = 14.952380 GB avg = 0.257 KB (by entry num) total key len = 1.504135 GB avg = 0.026 KB total ukey len = 1.038066 GB avg = 0.018 KB total ukey num = 0.058258567 Billion total entry num = 0.058258567 Billion write speed all = 37.387358960 MB/sec (with seq num) write speed all = 36.328503403 MB/sec (without seq num) 2024/11/19-15:25:57.136951 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-01 2024/11/19-15:25:57.137036 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00229/att-01/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:25:57.153342 139850978166528 [compaction_job.cc:1948] [default] [JOB 232] Generated table #5216: 20 keys, 3899 bytes, temperature: kUnknown 2024/11/19-15:25:57.153394 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001157153370, "cf_name": "default", "job": 232, "event": "table_file_creation", "file_number": 5216, "file_size": 3899, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 2576, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 320, "raw_average_key_size": 16, "raw_value_size": 2300, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 20, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-ZipOffsetBlobStore", "creation_time": 1732001157, "oldest_key_time": 0, "file_creation_time": 1732001157, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5216, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:57.153510 139850978166528 [compaction_job.cc:703] job-00232: subcompact[0], size: 0.000004 G, files: /storage/mytopling/.rocksdb [005216] 2024/11/19-15:25:57.155709 139850978166528 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 11466549 2024/11/19-15:25:57.155825 139850978166528 (Original Log Time 2024/11/19-15:25:57.109376) EVENT_LOG_v1 {"time_micros": 1732001157109361, "job": 232, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [5169], "score": 3.13857, "input_data_size": 3971, "oldest_snapshot_seqno": 147986157} 2024/11/19-15:25:57.155831 139850978166528 (Original Log Time 2024/11/19-15:25:57.155386) [compaction_job.cc:2022] [default] [JOB 232] Compacted 1@2 files to L3 => 3899 bytes 2024/11/19-15:25:57.155834 139850978166528 (Original Log Time 2024/11/19-15:25:57.155680) [compaction_job.cc:1224] [default] compacted to: files[5 69 86 56 0 0 0] max score 3.06 (136 files need compaction), MB/sec: 0.1 rd, 0.1 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 20, records dropped: 0 output_compression: Snappy 2024/11/19-15:25:57.155851 139850978166528 (Original Log Time 2024/11/19-15:25:57.155702) EVENT_LOG_v1 {"time_micros": 1732001157155691, "cf": "default", "job": 232, "event": "compaction_finished", "compaction_time_micros": 44140, "compaction_time_cpu_micros": 1522, "output_level": 3, "num_output_files": 1, "total_output_size": 3899, "num_input_records": 20, "num_output_records": 20, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 69, 86, 56, 0, 0, 0]} 2024/11/19-15:25:57.155940 139850978166528 [compaction_job.cc:2437] [default] [JOB 234] Compacting 1@2 files to L3, score 3.06, subcompactions 1 : 1 2024/11/19-15:25:57.155948 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 232 Base level 2, inputs: [5178(48MB)] 2024/11/19-15:25:57.166413 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005215.sst): second pass time = 0.00's,293817241.071'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1043.810'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1516.638'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.357'MB/sec write SST data time = 0.02's, 3683.093'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 293817241.071'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64720 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.92 avg-zval = 1020.17 usrkeys = 64720 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.92 avg-zval = 1020.17 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0470 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 15.018195 GB avg = 0.257 KB (by entry num) total key len = 1.505429 GB avg = 0.026 KB total ukey len = 1.038843 GB avg = 0.018 KB total ukey num = 0.058323287 Billion total entry num = 0.058323287 Billion write speed all = 37.536777711 MB/sec (with seq num) write speed all = 36.476831886 MB/sec (without seq num) 2024/11/19-15:25:57.170462 139851127064320 [compaction_job.cc:1948] [default] [JOB 227] Generated table #5215: 64720 keys, 66137412 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:57.170520 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001157170491, "cf_name": "default", "job": 227, "event": "table_file_creation", "file_number": 5215, "file_size": 66137412, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025616, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294400, "raw_average_key_size": 20, "raw_value_size": 65815062, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64720, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001156, "oldest_key_time": 0, "file_creation_time": 1732001156, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5215, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:57.187243 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005217.sst) /storage/mytopling/..//tmp/Topling-82038-Vap0ze 2024/11/19-15:25:57.242589 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005217.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:25:57.242698 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005217.sst): index pass time = 0.00's,6571.266'MB/sec index type = UintIndex_AllOne usrkeys = 63577 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:25:57.311632 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005217.sst): first pass time = 0.12's, 530.115'MB/sec 2024/11/19-15:25:57.311684 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005217.sst): old prealloc_size = 73819750, real_size = 64883608 2024/11/19-15:25:57.311708 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005217.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63577 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1017.04 avg-zval =1020.30 2024/11/19-15:25:57.365724 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005217.sst): second pass time = 0.00's,252580156.250'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1200.757'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.058'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.410'MB/sec write SST data time = 0.02's, 4258.064'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 252580156.250'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63577 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.04 avg-zval = 1020.30 usrkeys = 63577 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.04 avg-zval = 1020.30 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0000 GB value = 0.0649 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index =3973.5625 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 15.082855 GB avg = 0.258 KB (by entry num) total key len = 1.506701 GB avg = 0.026 KB total ukey len = 1.039606 GB avg = 0.018 KB total ukey num = 0.058386864 Billion total entry num = 0.058386864 Billion write speed all = 37.669500529 MB/sec (with seq num) write speed all = 36.608879487 MB/sec (without seq num) 2024/11/19-15:25:57.369502 139851127064320 [compaction_job.cc:1948] [default] [JOB 227] Generated table #5217: 63577 keys, 64868682 bytes (need compaction), temperature: kUnknown 2024/11/19-15:25:57.369553 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001157369530, "cf_name": "default", "job": 227, "event": "table_file_creation", "file_number": 5217, "file_size": 64868682, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64867360, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1271540, "raw_average_key_size": 20, "raw_value_size": 64660520, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63577, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001157, "oldest_key_time": 0, "file_creation_time": 1732001157, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5217, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:25:57.383853 139851127064320 [compaction_job.cc:703] job-00227: subcompact[0], size: 0.131006 G, files: /storage/mytopling/.rocksdb [005215,005217] 2024/11/19-15:25:57.384570 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 11821184 2024/11/19-15:25:57.384918 139851127064320 (Original Log Time 2024/11/19-15:25:56.965204) EVENT_LOG_v1 {"time_micros": 1732001156965190, "job": 227, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5065, 5066, 5068, 5069], "score": 4.27649, "input_data_size": 134099264, "oldest_snapshot_seqno": -1} 2024/11/19-15:25:57.384942 139851127064320 (Original Log Time 2024/11/19-15:25:57.384232) [compaction_job.cc:2022] [default] [JOB 227] Compacted 4@1 files to L2 => 131006094 bytes 2024/11/19-15:25:57.384946 139851127064320 (Original Log Time 2024/11/19-15:25:57.384539) [compaction_job.cc:1224] [default] compacted to: files[5 65 88 56 0 0 0] max score 3.11 (137 files need compaction), MB/sec: 320.3 rd, 312.9 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(124.9 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128297, records dropped: 0 output_compression: Snappy 2024/11/19-15:25:57.384965 139851127064320 (Original Log Time 2024/11/19-15:25:57.384563) EVENT_LOG_v1 {"time_micros": 1732001157384552, "cf": "default", "job": 227, "event": "compaction_finished", "compaction_time_micros": 418662, "compaction_time_cpu_micros": 414837, "output_level": 2, "num_output_files": 2, "total_output_size": 131006094, "num_input_records": 128297, "num_output_records": 128297, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 65, 88, 56, 0, 0, 0]} 2024/11/19-15:25:57.388993 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005020.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:57.389018 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001157389015, "job": 227, "event": "table_file_deletion", "file_number": 5020} 2024/11/19-15:25:57.392904 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005022.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:57.392936 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001157392933, "job": 227, "event": "table_file_deletion", "file_number": 5022} 2024/11/19-15:25:57.396641 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005025.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:57.396661 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001157396659, "job": 227, "event": "table_file_deletion", "file_number": 5025} 2024/11/19-15:25:57.400314 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005027.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:25:57.400333 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001157400331, "job": 227, "event": "table_file_deletion", "file_number": 5027} 2024/11/19-15:25:57.400442 139851127064320 [compaction_job.cc:2437] [default] [JOB 235] Compacting 1@2 files to L3, score 3.11, subcompactions 1 : 1 2024/11/19-15:25:57.400449 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 233 Base level 2, inputs: [5179(33MB)] 2024/11/19-15:25:57.453054 139850978166528 [dcompact_etcd.cc:1163] job-00234/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.297 sec, input = 49.413 MiB 2024/11/19-15:25:57.726594 139851127064320 [dcompact_etcd.cc:1163] job-00235/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.326 sec, input = 40.653 MiB 2024/11/19-15:25:58.694493 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-00 2024/11/19-15:25:58.694596 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00234/att-00/compact.done) = No such file or directory: 1.235 sec on http probe failed: 1, retry 2024/11/19-15:25:58.752152 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-00 2024/11/19-15:25:58.752242 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00235/att-00/compact.done) = No such file or directory: 1.016 sec on http probe failed: 1, retry 2024/11/19-15:25:58.766277 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-00 2024/11/19-15:25:58.766360 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00230/att-00/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:25:58.802374 139852349699840 [db_impl_write.cc:2181] [default] New memtable created with log file: #5218. Immutable memtables: 0. 2024/11/19-15:25:58.802759 139852349699840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 11466548 2024/11/19-15:25:58.802845 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 236] Syncing log #5191 2024/11/19-15:25:58.802939 139850967680768 (Original Log Time 2024/11/19-15:25:58.802833) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:25:58.802946 139850967680768 [flush_job.cc:879] [default] [JOB 236] Flushing memtable with next log file: 5218 2024/11/19-15:25:58.802967 139850967680768 EVENT_LOG_v1 {"time_micros": 1732001158802962, "job": 236, "event": "flush_started", "num_memtables": 1, "num_entries": 1374636, "num_deletes": 0, "total_data_size": 454822188, "memory_usage": 537394568, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:25:58.802970 139850967680768 [flush_job.cc:910] [default] [JOB 236] Level-0 flush table #5219: started 2024/11/19-15:25:58.803415 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000030.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/005219.sst in ConvertToSST, mem_size = 512.499 M, time = 435.988 us 2024/11/19-15:25:59.121853 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-02 2024/11/19-15:25:59.122006 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00228/att-02/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:25:59.128906 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-00 2024/11/19-15:25:59.129019 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00231/att-00/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:25:59.142492 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-01 2024/11/19-15:25:59.142579 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00229/att-01/compact.done) = No such file or directory: 11.205 sec on http probe failed: 5, giveup 2024/11/19-15:25:59.142610 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00229/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.980 sec, real 11.467 (submit 256.424 ms wait 11.205 s), input 126.980 MiB, send shutdown 2024/11/19-15:25:59.151701 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-01 2024/11/19-15:25:59.227899 139851137550080 [dcompact_etcd.cc:1773] job-00229/att-01: CleanFiles 0.076 sec : 0.000 2024/11/19-15:25:59.248490 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/005219.sst): fsize = 512.501 M, time(ms): open: 0.050, seek: 0.015, finish: 0.033, meta: 0.013, Flush: 0.012, sync: 444.927, close: 0.010, all: 445.060 2024/11/19-15:25:59.249605 139850967680768 [flush_job.cc:1083] [default] [JOB 236] Flush lasted 446666 microseconds, and 264112 cpu microseconds. 2024/11/19-15:25:59.270301 139850967680768 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 11122551 2024/11/19-15:25:59.270380 139850967680768 (Original Log Time 2024/11/19-15:25:59.248529) [flush_job.cc:1029] [default] [JOB 236] Level-0 flush table #5219: 537395994 bytes OK (needs compaction) 2024/11/19-15:25:59.270387 139850967680768 (Original Log Time 2024/11/19-15:25:59.249620) [memtable_list.cc:560] [default] Level-0 commit table #5219 started 2024/11/19-15:25:59.270390 139850967680768 (Original Log Time 2024/11/19-15:25:59.270268) [memtable_list.cc:759] [default] Level-0 commit table #5219: memtable #1 done 2024/11/19-15:25:59.270393 139850967680768 (Original Log Time 2024/11/19-15:25:59.270294) EVENT_LOG_v1 {"time_micros": 1732001159270288, "job": 236, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [6, 65, 88, 56, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:25:59.270395 139850967680768 (Original Log Time 2024/11/19-15:25:59.270351) [db_impl_compaction_flush.cc:310] [default] Level summary: files[6 65 88 56 0 0 0] max score 3.08 (137 files need compaction) 2024/11/19-15:25:59.526578 139851137550080 [dcompact_etcd.cc:1163] job-00229/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.299 sec, input = 126.980 MiB 2024/11/19-15:25:59.574417 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-00 2024/11/19-15:25:59.574509 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00235/att-00/compact.done) = No such file or directory: 1.839 sec on http probe failed: 2, retry 2024/11/19-15:25:59.690393 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-00 2024/11/19-15:25:59.690494 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00234/att-00/compact.done) = No such file or directory: 2.230 sec on http probe failed: 2, retry 2024/11/19-15:26:00.396017 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-00 2024/11/19-15:26:00.396141 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00235/att-00/compact.done) = No such file or directory: 2.661 sec on http probe failed: 3, retry 2024/11/19-15:26:00.684717 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-00 2024/11/19-15:26:00.684796 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00234/att-00/compact.done) = No such file or directory: 3.226 sec on http probe failed: 3, retry 2024/11/19-15:26:00.774717 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-00 2024/11/19-15:26:00.774789 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00230/att-00/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:26:01.128174 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-02 2024/11/19-15:26:01.128247 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00228/att-02/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:26:01.133584 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-00 2024/11/19-15:26:01.133642 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00231/att-00/compact.done) = No such file or directory: 7.192 sec on http probe failed: 3, retry 2024/11/19-15:26:01.214802 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-00 2024/11/19-15:26:01.214864 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00235/att-00/compact.done) = No such file or directory: 3.483 sec on http probe failed: 4, retry 2024/11/19-15:26:01.680660 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-00 2024/11/19-15:26:01.680774 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00234/att-00/compact.done) = No such file or directory: 4.220 sec on http probe failed: 4, retry 2024/11/19-15:26:01.702162 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-00 2024/11/19-15:26:01.702262 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00233/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:26:02.038320 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-00 2024/11/19-15:26:02.038423 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00235/att-00/compact.done) = No such file or directory: 4.301 sec on http probe failed: 5, giveup 2024/11/19-15:26:02.038442 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00235/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 40.653 sec, real 4.638 (submit 325.726 ms wait 4.301 s), input 40.653 MiB, send shutdown 2024/11/19-15:26:02.046332 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-00 2024/11/19-15:26:02.046571 139851127064320 [dcompact_etcd.cc:1773] job-00235/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:02.327791 139851127064320 [dcompact_etcd.cc:1163] job-00235/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 40.653 MiB 2024/11/19-15:26:02.677093 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-00 2024/11/19-15:26:02.677197 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00234/att-00/compact.done) = No such file or directory: 5.216 sec on http probe failed: 5, giveup 2024/11/19-15:26:02.677217 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00234/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 49.413 sec, real 5.521 (submit 296.685 ms wait 5.216 s), input 49.413 MiB, send shutdown 2024/11/19-15:26:02.682020 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-00 2024/11/19-15:26:02.693638 139850978166528 [dcompact_etcd.cc:1773] job-00234/att-00: CleanFiles 0.012 sec : 0.000 2024/11/19-15:26:02.706301 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-02 2024/11/19-15:26:02.706387 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00229/att-02/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:26:02.779889 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-00 2024/11/19-15:26:02.779967 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00230/att-00/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:26:02.955549 139850978166528 [dcompact_etcd.cc:1163] job-00234/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.262 sec, input = 49.413 MiB 2024/11/19-15:26:03.133374 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-02 2024/11/19-15:26:03.133456 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00228/att-02/compact.done) = No such file or directory: 7.185 sec on http probe failed: 3, retry 2024/11/19-15:26:03.140801 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-00 2024/11/19-15:26:03.140864 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00231/att-00/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:26:03.353644 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-01 2024/11/19-15:26:03.353741 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00235/att-01/compact.done) = No such file or directory: 1.016 sec on http probe failed: 1, retry 2024/11/19-15:26:03.709989 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-00 2024/11/19-15:26:03.710067 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00233/att-00/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:26:04.173043 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-01 2024/11/19-15:26:04.173114 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00235/att-01/compact.done) = No such file or directory: 1.839 sec on http probe failed: 2, retry 2024/11/19-15:26:04.196537 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-01 2024/11/19-15:26:04.196605 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00234/att-01/compact.done) = No such file or directory: 1.235 sec on http probe failed: 1, retry 2024/11/19-15:26:04.712340 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-02 2024/11/19-15:26:04.712438 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00229/att-02/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:26:04.787824 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-00 2024/11/19-15:26:04.787886 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00230/att-00/compact.done) = No such file or directory: 11.200 sec on http probe failed: 5, giveup 2024/11/19-15:26:04.787903 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00230/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.896 sec, real 11.481 (submit 273.403 ms wait 11.200 s), input 126.896 MiB, send shutdown 2024/11/19-15:26:04.797355 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-00 2024/11/19-15:26:04.797574 139851148035840 [dcompact_etcd.cc:1773] job-00230/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:04.995616 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-01 2024/11/19-15:26:04.995707 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00235/att-01/compact.done) = No such file or directory: 2.658 sec on http probe failed: 3, retry 2024/11/19-15:26:05.083620 139851148035840 [dcompact_etcd.cc:1163] job-00230/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.286 sec, input = 126.896 MiB 2024/11/19-15:26:05.137939 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-02 2024/11/19-15:26:05.138004 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00228/att-02/compact.done) = No such file or directory: 9.191 sec on http probe failed: 4, retry 2024/11/19-15:26:05.148729 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-00 2024/11/19-15:26:05.148785 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00231/att-00/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:26:05.148802 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00231/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.998 sec, real 11.543 (submit 331.820 ms wait 11.204 s), input 126.998 MiB, send shutdown 2024/11/19-15:26:05.157369 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-00 2024/11/19-15:26:05.157592 139851158521600 [dcompact_etcd.cc:1773] job-00231/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:05.193188 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-01 2024/11/19-15:26:05.193239 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00234/att-01/compact.done) = No such file or directory: 2.229 sec on http probe failed: 2, retry 2024/11/19-15:26:05.440734 139851158521600 [dcompact_etcd.cc:1163] job-00231/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.283 sec, input = 126.998 MiB 2024/11/19-15:26:05.716540 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-00 2024/11/19-15:26:05.716634 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00233/att-00/compact.done) = No such file or directory: 9.017 sec on http probe failed: 3, retry 2024/11/19-15:26:05.816205 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-01 2024/11/19-15:26:05.816268 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00235/att-01/compact.done) = No such file or directory: 3.481 sec on http probe failed: 4, retry 2024/11/19-15:26:06.190375 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-01 2024/11/19-15:26:06.190453 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00234/att-01/compact.done) = No such file or directory: 3.226 sec on http probe failed: 3, retry 2024/11/19-15:26:06.634642 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-01 2024/11/19-15:26:06.634739 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00235/att-01/compact.done) = No such file or directory: 4.302 sec on http probe failed: 5, giveup 2024/11/19-15:26:06.634755 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00235/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 40.653 sec, real 4.588 (submit 281.144 ms wait 4.302 s), input 40.653 MiB, send shutdown 2024/11/19-15:26:06.643160 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-01 2024/11/19-15:26:06.643369 139851127064320 [dcompact_etcd.cc:1773] job-00235/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:06.720956 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-02 2024/11/19-15:26:06.721024 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00229/att-02/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:26:06.899194 139851127064320 [dcompact_etcd.cc:1163] job-00235/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.256 sec, input = 40.653 MiB 2024/11/19-15:26:07.146687 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-02 2024/11/19-15:26:07.146762 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00228/att-02/compact.done) = No such file or directory: 11.195 sec on http probe failed: 5, giveup 2024/11/19-15:26:07.146780 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00228/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.892 sec, real 11.462 (submit 257.871 ms wait 11.195 s), input 126.892 MiB, send shutdown 2024/11/19-15:26:07.151834 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00228/att-02 2024/11/19-15:26:07.152059 139851169007360 [dcompact_etcd.cc:1773] job-00228/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:07.154045 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00228) = success 2024/11/19-15:26:07.154054 139851169007360 [dcompact_etcd.cc:1773] job-00228/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:26:07.154073 139851169007360 (Original Log Time 2024/11/19-15:25:32.664702) EVENT_LOG_v1 {"time_micros": 1732001132664695, "job": 228, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5070, 5071, 5073, 5074], "score": 4.02671, "input_data_size": 134098560, "oldest_snapshot_seqno": 147587623} 2024/11/19-15:26:07.154080 139851169007360 [compaction_job.cc:2437] [default] [JOB 228] Compacting 4@1 files to L2, score 4.03, subcompactions 1 : 1 2024/11/19-15:26:07.154087 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 226 Base level 1, inputs: [5070(31MB) 5071(31MB) 5073(31MB) 5074(31MB)] 2024/11/19-15:26:07.154316 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005220.sst) /storage/mytopling/..//tmp/Topling-82038-3lKsRs 2024/11/19-15:26:07.187530 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-01 2024/11/19-15:26:07.187594 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00234/att-01/compact.done) = No such file or directory: 4.223 sec on http probe failed: 4, retry 2024/11/19-15:26:07.207548 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005220.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:26:07.210895 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005220.sst): index pass time = 0.00's, 155.568'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64715 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:26:07.274511 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005220.sst): first pass time = 0.12's, 558.404'MB/sec 2024/11/19-15:26:07.274557 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005220.sst): old prealloc_size = 73819750, real_size = 66152480 2024/11/19-15:26:07.274615 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005220.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64715 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.00 avg-zval =1020.26 2024/11/19-15:26:07.274667 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005220.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.067 GB 2024/11/19-15:26:07.334807 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005220.sst): second pass time = 0.00's,144332019.737'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1098.074'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2030.960'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.381'MB/sec write SST data time = 0.02's, 3729.176'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 144332019.737'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64715 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.26 usrkeys = 64715 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0465 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 15.148671 GB avg = 0.259 KB (by entry num) total key len = 1.507995 GB avg = 0.026 KB total ukey len = 1.040382 GB avg = 0.018 KB total ukey num = 0.058451579 Billion total entry num = 0.058451579 Billion write speed all = 36.984679385 MB/sec (with seq num) write speed all = 35.946386145 MB/sec (without seq num) 2024/11/19-15:26:07.338394 139851169007360 [compaction_job.cc:1948] [default] [JOB 228] Generated table #5220: 64715 keys, 66137732 bytes (need compaction), temperature: kUnknown 2024/11/19-15:26:07.338448 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001167338424, "cf_name": "default", "job": 228, "event": "table_file_creation", "file_number": 5220, "file_size": 66137732, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025936, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294300, "raw_average_key_size": 20, "raw_value_size": 65815401, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64715, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001167, "oldest_key_time": 0, "file_creation_time": 1732001167, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5220, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:07.352819 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005221.sst) /storage/mytopling/..//tmp/Topling-82038-oupDuo 2024/11/19-15:26:07.404041 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005221.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:26:07.407297 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005221.sst): index pass time = 0.00's, 157.746'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63594 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:26:07.469511 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005221.sst): first pass time = 0.12's, 565.212'MB/sec 2024/11/19-15:26:07.469562 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005221.sst): old prealloc_size = 73819750, real_size = 65005632 2024/11/19-15:26:07.469623 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005221.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63594 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1016.99 avg-zval =1020.24 2024/11/19-15:26:07.469678 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005221.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.066 GB 2024/11/19-15:26:07.527722 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005221.sst): second pass time = 0.00's,228531081.272'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1117.811'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1969.392'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.383'MB/sec write SST data time = 0.02's, 3923.912'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 228531081.272'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63594 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 usrkeys = 63594 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7925 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0430 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 15.213345 GB avg = 0.260 KB (by entry num) total key len = 1.509267 GB avg = 0.026 KB total ukey len = 1.041146 GB avg = 0.018 KB total ukey num = 0.058515173 Billion total entry num = 0.058515173 Billion write speed all = 37.115208903 MB/sec (with seq num) write speed all = 36.076231067 MB/sec (without seq num) 2024/11/19-15:26:07.531939 139851169007360 [compaction_job.cc:1948] [default] [JOB 228] Generated table #5221: 63594 keys, 64991108 bytes (need compaction), temperature: kUnknown 2024/11/19-15:26:07.531990 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001167531967, "cf_name": "default", "job": 228, "event": "table_file_creation", "file_number": 5221, "file_size": 64991108, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64881200, "index_size": 108352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1271880, "raw_average_key_size": 20, "raw_value_size": 64674296, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63594, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001167, "oldest_key_time": 0, "file_creation_time": 1732001167, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5221, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:07.546111 139851169007360 [compaction_job.cc:703] job-00228: subcompact[0], size: 0.131129 G, files: /storage/mytopling/.rocksdb [005220,005221] 2024/11/19-15:26:07.546859 139851169007360 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 11466547 2024/11/19-15:26:07.547020 139851169007360 (Original Log Time 2024/11/19-15:26:07.154115) EVENT_LOG_v1 {"time_micros": 1732001167154099, "job": 228, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5070, 5071, 5073, 5074], "score": 4.02671, "input_data_size": 134098560, "oldest_snapshot_seqno": 147587623} 2024/11/19-15:26:07.547027 139851169007360 (Original Log Time 2024/11/19-15:26:07.546529) [compaction_job.cc:2022] [default] [JOB 228] Compacted 4@1 files to L2 => 131128840 bytes 2024/11/19-15:26:07.547031 139851169007360 (Original Log Time 2024/11/19-15:26:07.546830) [compaction_job.cc:1224] [default] compacted to: files[6 61 90 56 0 0 0] max score 3.16 (139 files need compaction), MB/sec: 342.1 rd, 334.5 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.1 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128309, records dropped: 0 output_compression: Snappy 2024/11/19-15:26:07.547048 139851169007360 (Original Log Time 2024/11/19-15:26:07.546853) EVENT_LOG_v1 {"time_micros": 1732001167546841, "cf": "default", "job": 228, "event": "compaction_finished", "compaction_time_micros": 392007, "compaction_time_cpu_micros": 390269, "output_level": 2, "num_output_files": 2, "total_output_size": 131128840, "num_input_records": 128309, "num_output_records": 128309, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 61, 90, 56, 0, 0, 0]} 2024/11/19-15:26:07.550974 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005029.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:07.550999 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001167550997, "job": 228, "event": "table_file_deletion", "file_number": 5029} 2024/11/19-15:26:07.555368 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005031.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:07.555389 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001167555387, "job": 228, "event": "table_file_deletion", "file_number": 5031} 2024/11/19-15:26:07.559295 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005033.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:07.559316 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001167559314, "job": 228, "event": "table_file_deletion", "file_number": 5033} 2024/11/19-15:26:07.563203 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005035.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:07.563222 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001167563221, "job": 228, "event": "table_file_deletion", "file_number": 5035} 2024/11/19-15:26:07.563367 139851169007360 [compaction_job.cc:2437] [default] [JOB 237] Compacting 4@2 files to L3, score 3.16, subcompactions 1 : 1 2024/11/19-15:26:07.563376 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 235 Base level 2, inputs: [5185(52MB) 5186(52MB) 5189(52MB) 5190(52MB)] 2024/11/19-15:26:07.725682 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-00 2024/11/19-15:26:07.725763 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00233/att-00/compact.done) = No such file or directory: 11.023 sec on http probe failed: 4, retry 2024/11/19-15:26:07.879582 139851169007360 [dcompact_etcd.cc:1163] job-00237/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.316 sec, input = 255.640 MiB 2024/11/19-15:26:07.925695 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-02 2024/11/19-15:26:07.925763 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00235/att-02/compact.done) = No such file or directory: 1.016 sec on http probe failed: 1, retry 2024/11/19-15:26:08.184031 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-01 2024/11/19-15:26:08.184102 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00234/att-01/compact.done) = No such file or directory: 5.220 sec on http probe failed: 5, giveup 2024/11/19-15:26:08.184119 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00234/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 49.413 sec, real 5.490 (submit 261.820 ms wait 5.220 s), input 49.413 MiB, send shutdown 2024/11/19-15:26:08.191855 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-01 2024/11/19-15:26:08.192083 139850978166528 [dcompact_etcd.cc:1773] job-00234/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:08.265482 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-01 2024/11/19-15:26:08.265548 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00230/att-01/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:26:08.479221 139850978166528 [dcompact_etcd.cc:1163] job-00234/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.287 sec, input = 49.413 MiB 2024/11/19-15:26:08.624473 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-01 2024/11/19-15:26:08.624569 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00231/att-01/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:26:08.729818 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-02 2024/11/19-15:26:08.729885 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00229/att-02/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:26:08.744001 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-02 2024/11/19-15:26:08.744058 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00235/att-02/compact.done) = No such file or directory: 1.840 sec on http probe failed: 2, retry 2024/11/19-15:26:09.562051 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-02 2024/11/19-15:26:09.562123 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00235/att-02/compact.done) = No such file or directory: 2.658 sec on http probe failed: 3, retry 2024/11/19-15:26:09.723568 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-02 2024/11/19-15:26:09.723644 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00234/att-02/compact.done) = No such file or directory: 1.235 sec on http probe failed: 1, retry 2024/11/19-15:26:09.730290 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-00 2024/11/19-15:26:09.730344 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00233/att-00/compact.done) = No such file or directory: 13.032 sec on http probe failed: 5, giveup 2024/11/19-15:26:09.730361 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00233/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 254.473 sec, real 13.320 (submit 283.123 ms wait 13.032 s), input 254.473 MiB, send shutdown 2024/11/19-15:26:09.738061 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-00 2024/11/19-15:26:09.738263 139851179493120 [dcompact_etcd.cc:1773] job-00233/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:10.035067 139851179493120 [dcompact_etcd.cc:1163] job-00233/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.297 sec, input = 254.473 MiB 2024/11/19-15:26:10.273869 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-01 2024/11/19-15:26:10.273948 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00230/att-01/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:26:10.380735 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-02 2024/11/19-15:26:10.380796 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00235/att-02/compact.done) = No such file or directory: 3.476 sec on http probe failed: 4, retry 2024/11/19-15:26:10.633610 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-01 2024/11/19-15:26:10.633686 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00231/att-01/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:26:10.720591 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-02 2024/11/19-15:26:10.720656 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00234/att-02/compact.done) = No such file or directory: 2.233 sec on http probe failed: 2, retry 2024/11/19-15:26:10.738858 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-02 2024/11/19-15:26:10.738913 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00229/att-02/compact.done) = No such file or directory: 11.203 sec on http probe failed: 5, giveup 2024/11/19-15:26:10.738937 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00229/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.980 sec, real 11.511 (submit 298.567 ms wait 11.203 s), input 126.980 MiB, send shutdown 2024/11/19-15:26:10.747518 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00229/att-02 2024/11/19-15:26:10.748389 139851137550080 [dcompact_etcd.cc:1773] job-00229/att-02: CleanFiles 0.001 sec : 0.000 2024/11/19-15:26:10.750245 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00229) = success 2024/11/19-15:26:10.750256 139851137550080 [dcompact_etcd.cc:1773] job-00229/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:26:10.750275 139851137550080 (Original Log Time 2024/11/19-15:25:36.165391) EVENT_LOG_v1 {"time_micros": 1732001136165385, "job": 229, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5075, 5077, 5078, 5079], "score": 3.77693, "input_data_size": 134099968, "oldest_snapshot_seqno": 147627534} 2024/11/19-15:26:10.750282 139851137550080 [compaction_job.cc:2437] [default] [JOB 229] Compacting 4@1 files to L2, score 3.78, subcompactions 1 : 1 2024/11/19-15:26:10.750290 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 227 Base level 1, inputs: [5075(31MB) 5077(31MB) 5078(31MB) 5079(31MB)] 2024/11/19-15:26:10.750549 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005222.sst) /storage/mytopling/..//tmp/Topling-82038-oyueqC 2024/11/19-15:26:10.803858 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005222.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:26:10.804010 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005222.sst): index pass time = 0.00's,4293.083'MB/sec index type = UintIndex_AllOne usrkeys = 64715 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:26:10.870614 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005222.sst): first pass time = 0.12's, 558.995'MB/sec 2024/11/19-15:26:10.870665 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005222.sst): old prealloc_size = 73819750, real_size = 66041768 2024/11/19-15:26:10.870687 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005222.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64715 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.99 avg-zval =1020.25 2024/11/19-15:26:10.924703 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005222.sst): second pass time = 0.00's,133228198.381'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1222.242'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.317'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.416'MB/sec write SST data time = 0.02's, 4356.829'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 133228198.381'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64715 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.99 avg-zval = 1020.25 usrkeys = 64715 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.99 avg-zval = 1020.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.6875 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 15.279160 GB avg = 0.261 KB (by entry num) total key len = 1.510561 GB avg = 0.026 KB total ukey len = 1.041922 GB avg = 0.018 KB total ukey num = 0.058579888 Billion total entry num = 0.058579888 Billion write speed all = 36.985305088 MB/sec (with seq num) write speed all = 35.952961521 MB/sec (without seq num) 2024/11/19-15:26:10.928286 139851137550080 [compaction_job.cc:1948] [default] [JOB 229] Generated table #5222: 64715 keys, 66026570 bytes (need compaction), temperature: kUnknown 2024/11/19-15:26:10.928338 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001170928315, "cf_name": "default", "job": 229, "event": "table_file_creation", "file_number": 5222, "file_size": 66026570, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025264, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294300, "raw_average_key_size": 20, "raw_value_size": 65814730, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64715, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001170, "oldest_key_time": 0, "file_creation_time": 1732001170, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5222, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:10.942530 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005223.sst) /storage/mytopling/..//tmp/Topling-82038-5bDIkb 2024/11/19-15:26:10.993768 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005223.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:26:10.998992 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005223.sst): index pass time = 0.01's, 98.108'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63681 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:26:11.060115 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005223.sst): first pass time = 0.12's, 561.717'MB/sec 2024/11/19-15:26:11.060183 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005223.sst): old prealloc_size = 73819750, real_size = 65096848 2024/11/19-15:26:11.060257 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005223.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63681 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0648 GB zip-val = 0.0650 GB avg-val =1017.02 avg-zval =1020.28 2024/11/19-15:26:11.060333 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005223.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.066 GB 2024/11/19-15:26:11.119073 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005223.sst): second pass time = 0.00's,252988750.000'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1105.365'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1553.465'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.378'MB/sec write SST data time = 0.02's, 3897.575'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 252988750.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63681 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.02 avg-zval = 1020.28 usrkeys = 63681 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.02 avg-zval = 1020.28 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7935 UnZipSize{ index = 0.0008 GB value = 0.0648 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0650 GB dict = 0.00 MB all = 0.0651 GB } UnZip/Zip{ index = 7.0444 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 15.343925 GB avg = 0.262 KB (by entry num) total key len = 1.511835 GB avg = 0.026 KB total ukey len = 1.042686 GB avg = 0.018 KB total ukey num = 0.058643569 Billion total entry num = 0.058643569 Billion write speed all = 37.114887310 MB/sec (with seq num) write speed all = 36.081863813 MB/sec (without seq num) 2024/11/19-15:26:11.122597 139851137550080 [compaction_job.cc:1948] [default] [JOB 229] Generated table #5223: 63681 keys, 65082308 bytes (need compaction), temperature: kUnknown 2024/11/19-15:26:11.122650 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001171122627, "cf_name": "default", "job": 229, "event": "table_file_creation", "file_number": 5223, "file_size": 65082308, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64972288, "index_size": 108480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1273620, "raw_average_key_size": 20, "raw_value_size": 64765120, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63681, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001170, "oldest_key_time": 0, "file_creation_time": 1732001170, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5223, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:11.137369 139851137550080 [compaction_job.cc:703] job-00229: subcompact[0], size: 0.131109 G, files: /storage/mytopling/.rocksdb [005222,005223] 2024/11/19-15:26:11.138092 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 11821182 2024/11/19-15:26:11.138239 139851137550080 (Original Log Time 2024/11/19-15:26:10.750317) EVENT_LOG_v1 {"time_micros": 1732001170750303, "job": 229, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5075, 5077, 5078, 5079], "score": 3.77693, "input_data_size": 134099968, "oldest_snapshot_seqno": 147627534} 2024/11/19-15:26:11.138246 139851137550080 (Original Log Time 2024/11/19-15:26:11.137749) [compaction_job.cc:2022] [default] [JOB 229] Compacted 4@1 files to L2 => 131108878 bytes 2024/11/19-15:26:11.138249 139851137550080 (Original Log Time 2024/11/19-15:26:11.138061) [compaction_job.cc:1224] [default] compacted to: files[6 57 92 56 0 0 0] max score 3.08 (137 files need compaction), MB/sec: 346.5 rd, 338.7 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128396, records dropped: 0 output_compression: Snappy 2024/11/19-15:26:11.138266 139851137550080 (Original Log Time 2024/11/19-15:26:11.138085) EVENT_LOG_v1 {"time_micros": 1732001171138072, "cf": "default", "job": 229, "event": "compaction_finished", "compaction_time_micros": 387065, "compaction_time_cpu_micros": 385356, "output_level": 2, "num_output_files": 2, "total_output_size": 131108878, "num_input_records": 128396, "num_output_records": 128396, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 57, 92, 56, 0, 0, 0]} 2024/11/19-15:26:11.142366 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005036.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:11.142391 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001171142388, "job": 229, "event": "table_file_deletion", "file_number": 5036} 2024/11/19-15:26:11.146354 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005039.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:11.146375 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001171146373, "job": 229, "event": "table_file_deletion", "file_number": 5039} 2024/11/19-15:26:11.150430 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005041.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:11.150451 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001171150448, "job": 229, "event": "table_file_deletion", "file_number": 5041} 2024/11/19-15:26:11.154770 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005043.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:11.154791 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001171154789, "job": 229, "event": "table_file_deletion", "file_number": 5043} 2024/11/19-15:26:11.154913 139851137550080 [compaction_job.cc:2437] [default] [JOB 238] Compacting 4@2 files to L3, score 3.08, subcompactions 1 : 1 2024/11/19-15:26:11.154932 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 236 Base level 2, inputs: [5193(52MB) 5194(52MB) 5197(52MB) 5199(52MB)] 2024/11/19-15:26:11.203805 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-02 2024/11/19-15:26:11.203898 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00235/att-02/compact.done) = No such file or directory: 4.295 sec on http probe failed: 5, giveup 2024/11/19-15:26:11.203916 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00235/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 40.653 sec, real 4.560 (submit 255.751 ms wait 4.295 s), input 40.653 MiB, send shutdown 2024/11/19-15:26:11.209370 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00235/att-02 2024/11/19-15:26:11.209579 139851127064320 [dcompact_etcd.cc:1773] job-00235/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:11.211380 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00235) = success 2024/11/19-15:26:11.211387 139851127064320 [dcompact_etcd.cc:1773] job-00235/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:26:11.211405 139851127064320 (Original Log Time 2024/11/19-15:25:57.400461) EVENT_LOG_v1 {"time_micros": 1732001157400454, "job": 235, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [5179], "score": 3.10601, "input_data_size": 34637141, "oldest_snapshot_seqno": 148095780} 2024/11/19-15:26:11.211413 139851127064320 [compaction_job.cc:2437] [default] [JOB 235] Compacting 1@2 files to L3, score 3.11, subcompactions 1 : 1 2024/11/19-15:26:11.211419 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 233 Base level 2, inputs: [5179(33MB)] 2024/11/19-15:26:11.211667 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005224.sst) /storage/mytopling/..//tmp/Topling-82038-MmT12M 2024/11/19-15:26:11.366422 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005224.sst): first pass time = 0.15's, 275.470'MB/sec 2024/11/19-15:26:11.366422 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005224.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.043 GB 2024/11/19-15:26:11.366449 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005224.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, dictZip workingMem = 0.0019 GB, waited 0.000 sec, Key+Value = 0.043 GB 2024/11/19-15:26:11.366537 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005224.sst): index pass time = 0.00's,51041.205'MB/sec index type = UintIndex_AllOne usrkeys = 576000 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0046 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:26:11.525378 139851137550080 [dcompact_etcd.cc:1163] job-00238/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.370 sec, input = 255.642 MiB 2024/11/19-15:26:11.714776 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-02 2024/11/19-15:26:11.714852 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00234/att-02/compact.done) = No such file or directory: 3.230 sec on http probe failed: 3, retry 2024/11/19-15:26:11.899466 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005224.sst): old prealloc_size = 38100855, real_size = 19175544 2024/11/19-15:26:11.899509 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005224.sst): index type = UintIndex_AllOne , store type = DictZipBlobStore usrkeys = 576000 prefix = 4 raw-key = 0.0046 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0311 GB zip-val = 0.0190 GB avg-val = 54.01 avg-zval = 33.04 2024/11/19-15:26:11.943505 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005224.sst): second pass time = 0.53's, 58.364'MB/sec, value only(73.0% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.04's, 439.457'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 8.665'MB/sec (index lex order gen) rebuild zvType time = 0.01's, 14.869'MB/sec write SST data time = 0.03's, 563.760'MB/sec dict compress time = 0.00's, 1164.031'MB/sec z-dict build time = 0.00's, sample length = 0.313'MB, throughput = 3002.911'MB/sec zip my value time = 0.53's, unzip length = 0.031'GB zip my value throughput = 58.364'MB/sec zip pipeline throughput = 58.364'MB/sec entries = 576000 avg-key = 20.00 avg-zkey = 0.00 avg-val = 54.01 avg-zval = 33.04 usrkeys = 576000 avg-key = 12.00 avg-zkey = 0.00 avg-val = 54.01 avg-zval = 33.04 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 3 UnZipSize{ index = 0.0069 GB value = 0.0311 GB dict = 0.31 MB all = 0.0426 GB } __ZipSize{ index = 0.0000 GB value = 0.0190 GB dict = 0.31 MB all = 0.0193 GB } UnZip/Zip{ index =36000.0000 value = 1.6346 dict = 1.00 all = 2.2035 } Zip/UnZip{ index = 0.0000 value = 0.6118 dict = 1.00 all = 0.4538 } ---------------------------- total value len = 15.375032 GB avg = 0.260 KB (by entry num) total key len = 1.523355 GB avg = 0.026 KB total ukey len = 1.049598 GB avg = 0.018 KB total ukey num = 0.059219569 Billion total entry num = 0.059219569 Billion write speed all = 37.141325716 MB/sec (with seq num) write speed all = 36.100046073 MB/sec (without seq num) 2024/11/19-15:26:11.944326 139851127064320 [compaction_job.cc:1948] [default] [JOB 235] Generated table #5224: 576000 keys, 19345061 bytes, temperature: kUnknown 2024/11/19-15:26:11.944378 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001171944355, "cf_name": "default", "job": 235, "event": "table_file_creation", "file_number": 5224, "file_size": 19345061, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 19031168, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 11520000, "raw_average_key_size": 20, "raw_value_size": 31107508, "raw_average_value_size": 54, "num_data_blocks": 1, "num_entries": 576000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-DictZipBlobStore", "creation_time": 1732001171, "oldest_key_time": 0, "file_creation_time": 1732001171, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5224, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:11.951905 139851127064320 [compaction_job.cc:703] job-00235: subcompact[0], size: 0.019345 G, files: /storage/mytopling/.rocksdb [005224] 2024/11/19-15:26:11.952566 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 12186785 2024/11/19-15:26:11.952689 139851127064320 (Original Log Time 2024/11/19-15:26:11.211444) EVENT_LOG_v1 {"time_micros": 1732001171211430, "job": 235, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [5179], "score": 3.10601, "input_data_size": 34637141, "oldest_snapshot_seqno": 148095780} 2024/11/19-15:26:11.952698 139851127064320 (Original Log Time 2024/11/19-15:26:11.952257) [compaction_job.cc:2022] [default] [JOB 235] Compacted 1@2 files to L3 => 19345061 bytes 2024/11/19-15:26:11.952703 139851127064320 (Original Log Time 2024/11/19-15:26:11.952538) [compaction_job.cc:1224] [default] compacted to: files[6 57 91 57 0 0 0] max score 3.03 (133 files need compaction), MB/sec: 46.8 rd, 26.1 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(33.0, 0.0 +0.0 blob) out(18.4 +0.0 blob), read-write-amplify(1.6) write-amplify(0.6) OK, records in: 576000, records dropped: 0 output_compression: Snappy 2024/11/19-15:26:11.952727 139851127064320 (Original Log Time 2024/11/19-15:26:11.952559) EVENT_LOG_v1 {"time_micros": 1732001171952549, "cf": "default", "job": 235, "event": "compaction_finished", "compaction_time_micros": 740473, "compaction_time_cpu_micros": 564867, "output_level": 3, "num_output_files": 1, "total_output_size": 19345061, "num_input_records": 576000, "num_output_records": 576000, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 57, 91, 57, 0, 0, 0]} 2024/11/19-15:26:11.952842 139851127064320 [compaction_job.cc:2437] [default] [JOB 239] Compacting 4@1 files to L2, score 3.03, subcompactions 1 : 1 2024/11/19-15:26:11.952855 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 237 Base level 1, inputs: [5092(31MB) 5093(31MB) 5094(31MB) 5096(31MB)] 2024/11/19-15:26:12.256908 139851127064320 [dcompact_etcd.cc:1163] job-00239/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.304 sec, input = 126.834 MiB 2024/11/19-15:26:12.282503 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-01 2024/11/19-15:26:12.282568 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00230/att-01/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:26:12.638879 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-01 2024/11/19-15:26:12.638982 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00231/att-01/compact.done) = No such file or directory: 7.193 sec on http probe failed: 3, retry 2024/11/19-15:26:12.711388 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-02 2024/11/19-15:26:12.711468 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00234/att-02/compact.done) = No such file or directory: 4.224 sec on http probe failed: 4, retry 2024/11/19-15:26:12.885778 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-00 2024/11/19-15:26:12.885875 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00237/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:26:13.710708 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-02 2024/11/19-15:26:13.710784 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00234/att-02/compact.done) = No such file or directory: 5.221 sec on http probe failed: 5, giveup 2024/11/19-15:26:13.710802 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00234/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 49.413 sec, real 5.519 (submit 287.057 ms wait 5.221 s), input 49.413 MiB, send shutdown 2024/11/19-15:26:13.719145 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00234/att-02 2024/11/19-15:26:13.719358 139850978166528 [dcompact_etcd.cc:1773] job-00234/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:13.721107 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00234) = success 2024/11/19-15:26:13.721117 139850978166528 [dcompact_etcd.cc:1773] job-00234/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:26:13.721135 139850978166528 (Original Log Time 2024/11/19-15:25:57.155960) EVENT_LOG_v1 {"time_micros": 1732001157155954, "job": 234, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [5178], "score": 3.05557, "input_data_size": 50543257, "oldest_snapshot_seqno": 148065784} 2024/11/19-15:26:13.721141 139850978166528 [compaction_job.cc:2437] [default] [JOB 234] Compacting 1@2 files to L3, score 3.06, subcompactions 1 : 1 2024/11/19-15:26:13.721155 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 232 Base level 2, inputs: [5178(48MB)] 2024/11/19-15:26:13.721432 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005225.sst) /storage/mytopling/..//tmp/Topling-82038-iEdqiO 2024/11/19-15:26:13.771568 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005225.sst): first pass time = 0.05's,1033.659'MB/sec 2024/11/19-15:26:13.771573 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005225.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.052 GB 2024/11/19-15:26:13.771597 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005225.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0031 GB, waited 0.000 sec, Key+Value = 0.052 GB 2024/11/19-15:26:13.775768 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005225.sst): index pass time = 0.00's, 222.914'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 77431 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0009 GB zip-key = 0.0001 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:26:14.298178 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-01 2024/11/19-15:26:14.298282 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00230/att-01/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:26:14.572664 139850978166528 [top_zip_table_builder.cc:2752] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005225.sst): part-0: reject ZipStore ratio = 0.804 2024/11/19-15:26:14.644722 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-01 2024/11/19-15:26:14.644826 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00231/att-01/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:26:14.652721 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005225.sst): old prealloc_size = 55597582, real_size = 50187856 2024/11/19-15:26:14.652793 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005225.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 77431 prefix = 4 raw-key = 0.0009 GB zip-key = 0.0001 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0500 GB zip-val = 0.0500 GB avg-val = 645.15 avg-zval = 646.21 2024/11/19-15:26:14.652877 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005225.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.881 sec, Key+Value = 0.052 GB 2024/11/19-15:26:14.737581 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005225.sst): second pass time = 0.80's, 62.361'MB/sec, value only(96.4% of KV) wait indexing time = 0.08's, remap KeyValue time = 0.08's, 591.263'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2183.591'MB/sec (index lex order gen) rebuild zvType time = 0.07's, 0.288'MB/sec write SST data time = 0.02's, 2867.828'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = 0.000'MB/sec zip my value time = 0.80's, unzip length = 0.050'GB zip my value throughput = 62.361'MB/sec zip pipeline throughput = 62.361'MB/sec entries = 77431 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.15 avg-zval = 646.21 usrkeys = 77431 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.15 avg-zval = 646.21 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6103 UnZipSize{ index = 0.0012 GB value = 0.0500 GB dict = 0.00 MB all = 0.0518 GB } __ZipSize{ index = 0.0001 GB value = 0.0500 GB dict = 0.00 MB all = 0.0502 GB } UnZip/Zip{ index = 9.4061 value = 0.9984 dict = 0.00 all = 1.0327 } Zip/UnZip{ index = 0.1063 value = 1.0016 dict = 0.00 all = 0.9683 } ---------------------------- total value len = 15.424987 GB avg = 0.260 KB (by entry num) total key len = 1.525213 GB avg = 0.026 KB total ukey len = 1.050837 GB avg = 0.018 KB total ukey num = 0.059297000 Billion total entry num = 0.059297000 Billion write speed all = 37.027812381 MB/sec (with seq num) write speed all = 35.991535186 MB/sec (without seq num) 2024/11/19-15:26:14.740822 139850978166528 [compaction_job.cc:1948] [default] [JOB 234] Generated table #5225: 77431 keys, 50170192 bytes, temperature: kUnknown 2024/11/19-15:26:14.740871 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001174740848, "cf_name": "default", "job": 234, "event": "table_file_creation", "file_number": 5225, "file_size": 50170192, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 50036608, "index_size": 131712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1858344, "raw_average_key_size": 24, "raw_value_size": 49954449, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 77431, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732001173, "oldest_key_time": 0, "file_creation_time": 1732001173, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5225, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:14.756377 139850978166528 [compaction_job.cc:703] job-00234: subcompact[0], size: 0.050170 G, files: /storage/mytopling/.rocksdb [005225] 2024/11/19-15:26:14.756998 139850978166528 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 12563695 2024/11/19-15:26:14.757145 139850978166528 (Original Log Time 2024/11/19-15:26:13.721184) EVENT_LOG_v1 {"time_micros": 1732001173721168, "job": 234, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [5178], "score": 3.05557, "input_data_size": 50543257, "oldest_snapshot_seqno": 148065784} 2024/11/19-15:26:14.757155 139850978166528 (Original Log Time 2024/11/19-15:26:14.756674) [compaction_job.cc:2022] [default] [JOB 234] Compacted 1@2 files to L3 => 50170192 bytes 2024/11/19-15:26:14.757160 139850978166528 (Original Log Time 2024/11/19-15:26:14.756969) [compaction_job.cc:1224] [default] compacted to: files[6 57 90 58 0 0 0] max score 2.91 (129 files need compaction), MB/sec: 48.8 rd, 48.5 wr, level 3, files in(1, 0) out(1 +0 blob) MB in(48.2, 0.0 +0.0 blob) out(47.8 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 77431, records dropped: 0 output_compression: Snappy 2024/11/19-15:26:14.757186 139850978166528 (Original Log Time 2024/11/19-15:26:14.756991) EVENT_LOG_v1 {"time_micros": 1732001174756980, "cf": "default", "job": 234, "event": "compaction_finished", "compaction_time_micros": 1035204, "compaction_time_cpu_micros": 421431, "output_level": 3, "num_output_files": 1, "total_output_size": 50170192, "num_input_records": 77431, "num_output_records": 77431, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 57, 90, 58, 0, 0, 0]} 2024/11/19-15:26:14.757306 139850978166528 [compaction_job.cc:2437] [default] [JOB 240] Compacting 4@2 files to L3, score 2.91, subcompactions 1 : 1 2024/11/19-15:26:14.757320 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 238 Base level 2, inputs: [5198(52MB) 5200(52MB) 5195(52MB) 5196(23MB)] 2024/11/19-15:26:14.894967 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-00 2024/11/19-15:26:14.895058 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00237/att-00/compact.done) = No such file or directory: 7.006 sec on http probe failed: 2, retry 2024/11/19-15:26:15.040704 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-01 2024/11/19-15:26:15.040788 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00233/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:26:15.057005 139850978166528 [dcompact_etcd.cc:1163] job-00240/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.299 sec, input = 219.968 MiB 2024/11/19-15:26:15.436963 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-00 2024/11/19-15:26:15.437057 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00239/att-00/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:26:16.303932 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-01 2024/11/19-15:26:16.304011 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00230/att-01/compact.done) = No such file or directory: 11.215 sec on http probe failed: 5, giveup 2024/11/19-15:26:16.304029 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00230/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.896 sec, real 11.506 (submit 285.975 ms wait 11.215 s), input 126.896 MiB, send shutdown 2024/11/19-15:26:16.309545 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-01 2024/11/19-15:26:16.309749 139851148035840 [dcompact_etcd.cc:1773] job-00230/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:16.530670 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-00 2024/11/19-15:26:16.530750 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00238/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:26:16.590422 139851148035840 [dcompact_etcd.cc:1163] job-00230/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 126.896 MiB 2024/11/19-15:26:16.650992 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-01 2024/11/19-15:26:16.651046 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00231/att-01/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:26:16.651061 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00231/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.998 sec, real 11.493 (submit 283.069 ms wait 11.204 s), input 126.998 MiB, send shutdown 2024/11/19-15:26:16.658456 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-01 2024/11/19-15:26:16.658658 139851158521600 [dcompact_etcd.cc:1773] job-00231/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:16.901346 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-00 2024/11/19-15:26:16.901404 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00237/att-00/compact.done) = No such file or directory: 9.016 sec on http probe failed: 3, retry 2024/11/19-15:26:16.947297 139851158521600 [dcompact_etcd.cc:1163] job-00231/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 126.998 MiB 2024/11/19-15:26:17.048680 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-01 2024/11/19-15:26:17.048739 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00233/att-01/compact.done) = No such file or directory: 7.006 sec on http probe failed: 2, retry 2024/11/19-15:26:17.442285 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-00 2024/11/19-15:26:17.442379 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00239/att-00/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:26:18.538626 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-00 2024/11/19-15:26:18.538702 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00238/att-00/compact.done) = No such file or directory: 7.005 sec on http probe failed: 2, retry 2024/11/19-15:26:18.910375 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-00 2024/11/19-15:26:18.910445 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00237/att-00/compact.done) = No such file or directory: 11.022 sec on http probe failed: 4, retry 2024/11/19-15:26:19.057941 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-01 2024/11/19-15:26:19.058002 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00233/att-01/compact.done) = No such file or directory: 9.014 sec on http probe failed: 3, retry 2024/11/19-15:26:19.349583 139852362282752 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 12186784 2024/11/19-15:26:19.450349 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-00 2024/11/19-15:26:19.450423 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00239/att-00/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:26:19.772638 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-02 2024/11/19-15:26:19.772731 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00230/att-02/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:26:20.062484 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-00 2024/11/19-15:26:20.062560 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00240/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:26:20.130142 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-02 2024/11/19-15:26:20.130207 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00231/att-02/compact.done) = No such file or directory: 3.175 sec on http probe failed: 1, retry 2024/11/19-15:26:20.547743 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-00 2024/11/19-15:26:20.547842 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00238/att-00/compact.done) = No such file or directory: 9.013 sec on http probe failed: 3, retry 2024/11/19-15:26:20.918875 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-00 2024/11/19-15:26:20.918966 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00237/att-00/compact.done) = No such file or directory: 13.031 sec on http probe failed: 5, giveup 2024/11/19-15:26:20.918985 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00237/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.640 sec, real 13.355 (submit 315.792 ms wait 13.031 s), input 255.640 MiB, send shutdown 2024/11/19-15:26:20.924755 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-00 2024/11/19-15:26:20.924991 139851169007360 [dcompact_etcd.cc:1773] job-00237/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:21.066406 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-01 2024/11/19-15:26:21.066483 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00233/att-01/compact.done) = No such file or directory: 11.023 sec on http probe failed: 4, retry 2024/11/19-15:26:21.210385 139851169007360 [dcompact_etcd.cc:1163] job-00237/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.285 sec, input = 255.640 MiB 2024/11/19-15:26:21.455800 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-00 2024/11/19-15:26:21.455860 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00239/att-00/compact.done) = No such file or directory: 9.194 sec on http probe failed: 4, retry 2024/11/19-15:26:21.778136 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-02 2024/11/19-15:26:21.778205 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00230/att-02/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:26:22.070927 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-00 2024/11/19-15:26:22.071023 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00240/att-00/compact.done) = No such file or directory: 7.006 sec on http probe failed: 2, retry 2024/11/19-15:26:22.138379 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-02 2024/11/19-15:26:22.138458 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00231/att-02/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:26:22.552801 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-00 2024/11/19-15:26:22.552884 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00238/att-00/compact.done) = No such file or directory: 11.023 sec on http probe failed: 4, retry 2024/11/19-15:26:23.071620 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-01 2024/11/19-15:26:23.071714 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00233/att-01/compact.done) = No such file or directory: 13.031 sec on http probe failed: 5, giveup 2024/11/19-15:26:23.071734 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00233/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 254.473 sec, real 13.333 (submit 296.719 ms wait 13.031 s), input 254.473 MiB, send shutdown 2024/11/19-15:26:23.079300 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-01 2024/11/19-15:26:23.079515 139851179493120 [dcompact_etcd.cc:1773] job-00233/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:23.377915 139851179493120 [dcompact_etcd.cc:1163] job-00233/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.298 sec, input = 254.473 MiB 2024/11/19-15:26:23.464888 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-00 2024/11/19-15:26:23.464967 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00239/att-00/compact.done) = No such file or directory: 11.199 sec on http probe failed: 5, giveup 2024/11/19-15:26:23.464984 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00239/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.834 sec, real 11.512 (submit 303.548 ms wait 11.199 s), input 126.834 MiB, send shutdown 2024/11/19-15:26:23.469631 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-00 2024/11/19-15:26:23.469838 139851127064320 [dcompact_etcd.cc:1773] job-00239/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:23.775191 139851127064320 [dcompact_etcd.cc:1163] job-00239/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.305 sec, input = 126.834 MiB 2024/11/19-15:26:23.787450 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-02 2024/11/19-15:26:23.787506 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00230/att-02/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:26:24.079884 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-00 2024/11/19-15:26:24.079966 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00240/att-00/compact.done) = No such file or directory: 9.014 sec on http probe failed: 3, retry 2024/11/19-15:26:24.147145 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-02 2024/11/19-15:26:24.147229 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00231/att-02/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:26:24.559486 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-00 2024/11/19-15:26:24.559564 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00238/att-00/compact.done) = No such file or directory: 13.028 sec on http probe failed: 5, giveup 2024/11/19-15:26:24.559582 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00238/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.642 sec, real 13.404 (submit 369.992 ms wait 13.028 s), input 255.642 MiB, send shutdown 2024/11/19-15:26:24.568390 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-00 2024/11/19-15:26:24.568595 139851137550080 [dcompact_etcd.cc:1773] job-00238/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:24.840813 139851137550080 [dcompact_etcd.cc:1163] job-00238/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.272 sec, input = 255.642 MiB 2024/11/19-15:26:25.797699 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-02 2024/11/19-15:26:25.797795 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00230/att-02/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:26:26.088598 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-00 2024/11/19-15:26:26.088687 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00240/att-00/compact.done) = No such file or directory: 11.023 sec on http probe failed: 4, retry 2024/11/19-15:26:26.155991 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-02 2024/11/19-15:26:26.156056 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00231/att-02/compact.done) = No such file or directory: 9.200 sec on http probe failed: 4, retry 2024/11/19-15:26:26.219630 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-01 2024/11/19-15:26:26.219694 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00237/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:26:26.954647 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-01 2024/11/19-15:26:26.954757 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00239/att-01/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:26:27.803323 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-02 2024/11/19-15:26:27.803431 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00230/att-02/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:26:27.803456 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00230/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.896 sec, real 11.494 (submit 280.604 ms wait 11.207 s), input 126.896 MiB, send shutdown 2024/11/19-15:26:27.807982 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00230/att-02 2024/11/19-15:26:27.808231 139851148035840 [dcompact_etcd.cc:1773] job-00230/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:27.810137 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00230) = success 2024/11/19-15:26:27.810150 139851148035840 [dcompact_etcd.cc:1773] job-00230/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:26:27.810173 139851148035840 (Original Log Time 2024/11/19-15:25:53.306064) EVENT_LOG_v1 {"time_micros": 1732001153306056, "job": 230, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5081, 5082, 5083, 5085], "score": 3.52715, "input_data_size": 134100032, "oldest_snapshot_seqno": 147924203} 2024/11/19-15:26:27.810184 139851148035840 [compaction_job.cc:2437] [default] [JOB 230] Compacting 4@1 files to L2, score 3.53, subcompactions 1 : 1 2024/11/19-15:26:27.810196 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 228 Base level 1, inputs: [5081(31MB) 5082(31MB) 5083(31MB) 5085(31MB)] 2024/11/19-15:26:27.810456 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005226.sst) /storage/mytopling/..//tmp/Topling-82038-ozHO2M 2024/11/19-15:26:27.896850 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005226.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:26:27.900162 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005226.sst): index pass time = 0.00's, 158.348'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64715 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:26:27.969649 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005226.sst): first pass time = 0.16's, 421.604'MB/sec 2024/11/19-15:26:27.969703 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005226.sst): old prealloc_size = 73819750, real_size = 66152352 2024/11/19-15:26:27.969758 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005226.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64715 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.00 avg-zval =1020.26 2024/11/19-15:26:27.969799 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005226.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.073 sec, Key+Value = 0.067 GB 2024/11/19-15:26:28.033984 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005226.sst): second pass time = 0.00's,232562819.788'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1029.183'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2163.274'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.343'MB/sec write SST data time = 0.02's, 3892.092'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 232562819.788'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64715 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.26 usrkeys = 64715 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0465 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 15.490802 GB avg = 0.261 KB (by entry num) total key len = 1.526507 GB avg = 0.026 KB total ukey len = 1.051614 GB avg = 0.018 KB total ukey num = 0.059361715 Billion total entry num = 0.059361715 Billion write speed all = 36.125121011 MB/sec (with seq num) write speed all = 35.116994949 MB/sec (without seq num) 2024/11/19-15:26:28.052622 139851148035840 [compaction_job.cc:1948] [default] [JOB 230] Generated table #5226: 64715 keys, 66137604 bytes (need compaction), temperature: kUnknown 2024/11/19-15:26:28.052676 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001188052653, "cf_name": "default", "job": 230, "event": "table_file_creation", "file_number": 5226, "file_size": 66137604, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025808, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294300, "raw_average_key_size": 20, "raw_value_size": 65815278, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64715, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001187, "oldest_key_time": 0, "file_creation_time": 1732001187, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5226, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:28.093749 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-00 2024/11/19-15:26:28.093843 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00240/att-00/compact.done) = No such file or directory: 13.032 sec on http probe failed: 5, giveup 2024/11/19-15:26:28.093862 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00240/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 219.968 sec, real 13.336 (submit 299.210 ms wait 13.032 s), input 219.968 MiB, send shutdown 2024/11/19-15:26:28.102469 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005227.sst) /storage/mytopling/..//tmp/Topling-82038-PWMDeG 2024/11/19-15:26:28.103066 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-00 2024/11/19-15:26:28.103300 139850978166528 [dcompact_etcd.cc:1773] job-00240/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:28.154864 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005227.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:26:28.154988 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005227.sst): index pass time = 0.00's,5800.794'MB/sec index type = UintIndex_AllOne usrkeys = 63597 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:26:28.160996 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-02 2024/11/19-15:26:28.161065 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00231/att-02/compact.done) = No such file or directory: 11.209 sec on http probe failed: 5, giveup 2024/11/19-15:26:28.161083 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00231/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.998 sec, real 11.502 (submit 288.563 ms wait 11.209 s), input 126.998 MiB, send shutdown 2024/11/19-15:26:28.170415 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00231/att-02 2024/11/19-15:26:28.170624 139851158521600 [dcompact_etcd.cc:1773] job-00231/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:28.172834 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00231) = success 2024/11/19-15:26:28.172843 139851158521600 [dcompact_etcd.cc:1773] job-00231/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:26:28.172861 139851158521600 (Original Log Time 2024/11/19-15:25:53.604894) EVENT_LOG_v1 {"time_micros": 1732001153604887, "job": 231, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5086, 5087, 5089, 5090], "score": 3.27737, "input_data_size": 134099264, "oldest_snapshot_seqno": 147924203} 2024/11/19-15:26:28.172867 139851158521600 [compaction_job.cc:2437] [default] [JOB 231] Compacting 4@1 files to L2, score 3.28, subcompactions 1 : 1 2024/11/19-15:26:28.172889 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 229 Base level 1, inputs: [5086(31MB) 5087(31MB) 5089(31MB) 5090(31MB)] 2024/11/19-15:26:28.173120 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005228.sst) /storage/mytopling/..//tmp/Topling-82038-0tEp9t 2024/11/19-15:26:28.222154 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005227.sst): first pass time = 0.12's, 551.116'MB/sec 2024/11/19-15:26:28.222205 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005227.sst): old prealloc_size = 73819750, real_size = 64901768 2024/11/19-15:26:28.222227 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005227.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63597 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:26:28.228124 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-01 2024/11/19-15:26:28.228218 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00237/att-01/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:26:28.229470 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005228.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:26:28.233174 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005228.sst): index pass time = 0.00's, 141.005'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64714 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:26:28.277088 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005227.sst): second pass time = 0.00's,235194945.455'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1182.674'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.601'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.404'MB/sec write SST data time = 0.02's, 4198.158'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 235194945.455'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63597 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 63597 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0000 GB value = 0.0649 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index =3974.8125 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 15.555481 GB avg = 0.262 KB (by entry num) total key len = 1.527779 GB avg = 0.026 KB total ukey len = 1.052377 GB avg = 0.018 KB total ukey num = 0.059425312 Billion total entry num = 0.059425312 Billion write speed all = 36.246418079 MB/sec (with seq num) write speed all = 35.237732519 MB/sec (without seq num) 2024/11/19-15:26:28.292932 139851148035840 [compaction_job.cc:1948] [default] [JOB 230] Generated table #5227: 63597 keys, 64886858 bytes (need compaction), temperature: kUnknown 2024/11/19-15:26:28.292990 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001188292965, "cf_name": "default", "job": 230, "event": "table_file_creation", "file_number": 5227, "file_size": 64886858, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64885520, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1271940, "raw_average_key_size": 20, "raw_value_size": 64678610, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63597, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001188, "oldest_key_time": 0, "file_creation_time": 1732001188, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5227, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:28.298165 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005228.sst): first pass time = 0.13's, 536.730'MB/sec 2024/11/19-15:26:28.298212 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005228.sst): old prealloc_size = 73819750, real_size = 66151744 2024/11/19-15:26:28.298270 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005228.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64714 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:26:28.298308 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005228.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.069 sec, Key+Value = 0.067 GB 2024/11/19-15:26:28.308997 139851148035840 [compaction_job.cc:703] job-00230: subcompact[0], size: 0.131024 G, files: /storage/mytopling/.rocksdb [005226,005227] 2024/11/19-15:26:28.310431 139851148035840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 12563694 2024/11/19-15:26:28.310707 139851148035840 (Original Log Time 2024/11/19-15:26:27.810233) EVENT_LOG_v1 {"time_micros": 1732001187810213, "job": 230, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5081, 5082, 5083, 5085], "score": 3.52715, "input_data_size": 134100032, "oldest_snapshot_seqno": 147924203} 2024/11/19-15:26:28.310721 139851148035840 (Original Log Time 2024/11/19-15:26:28.309748) [compaction_job.cc:2022] [default] [JOB 230] Compacted 4@1 files to L2 => 131024462 bytes 2024/11/19-15:26:28.310728 139851148035840 (Original Log Time 2024/11/19-15:26:28.310359) [compaction_job.cc:1224] [default] compacted to: files[6 53 92 58 0 0 0] max score 2.85 (127 files need compaction), MB/sec: 268.8 rd, 262.7 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128312, records dropped: 0 output_compression: Snappy 2024/11/19-15:26:28.310764 139851148035840 (Original Log Time 2024/11/19-15:26:28.310416) EVENT_LOG_v1 {"time_micros": 1732001188310389, "cf": "default", "job": 230, "event": "compaction_finished", "compaction_time_micros": 498800, "compaction_time_cpu_micros": 431952, "output_level": 2, "num_output_files": 2, "total_output_size": 131024462, "num_input_records": 128312, "num_output_records": 128312, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 53, 92, 58, 0, 0, 0]} 2024/11/19-15:26:28.323187 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005045.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:28.323215 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001188323212, "job": 230, "event": "table_file_deletion", "file_number": 5045} 2024/11/19-15:26:28.344958 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005050.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:28.344990 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001188344986, "job": 230, "event": "table_file_deletion", "file_number": 5050} 2024/11/19-15:26:28.350161 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005051.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:28.350184 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001188350181, "job": 230, "event": "table_file_deletion", "file_number": 5051} 2024/11/19-15:26:28.354482 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005052.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:28.354503 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001188354501, "job": 230, "event": "table_file_deletion", "file_number": 5052} 2024/11/19-15:26:28.355121 139851148035840 [compaction_job.cc:2437] [default] [JOB 241] Compacting 4@2 files to L3, score 2.85, subcompactions 1 : 1 2024/11/19-15:26:28.355133 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 239 Base level 2, inputs: [4784(61MB) 4618(31MB) 4778(31MB) 4628(31MB)] 2024/11/19-15:26:28.367000 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005228.sst): second pass time = 0.00's,233385322.695'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 961.631'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2041.040'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.312'MB/sec write SST data time = 0.02's, 3906.671'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 233385322.695'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64714 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 64714 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0464 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 15.621295 GB avg = 0.263 KB (by entry num) total key len = 1.529074 GB avg = 0.026 KB total ukey len = 1.053154 GB avg = 0.018 KB total ukey num = 0.059490026 Billion total entry num = 0.059490026 Billion write speed all = 36.381865318 MB/sec (with seq num) write speed all = 35.372273919 MB/sec (without seq num) 2024/11/19-15:26:28.370567 139851158521600 [compaction_job.cc:1948] [default] [JOB 231] Generated table #5228: 64714 keys, 66136964 bytes (need compaction), temperature: kUnknown 2024/11/19-15:26:28.370611 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001188370588, "cf_name": "default", "job": 231, "event": "table_file_creation", "file_number": 5228, "file_size": 66136964, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025200, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294280, "raw_average_key_size": 20, "raw_value_size": 65814661, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64714, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001188, "oldest_key_time": 0, "file_creation_time": 1732001188, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5228, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:28.385485 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005229.sst) /storage/mytopling/..//tmp/Topling-82038-DKzNFA 2024/11/19-15:26:28.386009 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-02 2024/11/19-15:26:28.386108 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00233/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:26:28.388608 139850978166528 [dcompact_etcd.cc:1163] job-00240/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.285 sec, input = 219.968 MiB 2024/11/19-15:26:28.440616 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005229.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:26:28.440703 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005229.sst): index pass time = 0.00's,9231.360'MB/sec index type = UintIndex_AllOne usrkeys = 63701 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:26:28.507190 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005229.sst): first pass time = 0.12's, 542.858'MB/sec 2024/11/19-15:26:28.507247 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005229.sst): old prealloc_size = 73819750, real_size = 65008040 2024/11/19-15:26:28.507272 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005229.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63701 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0648 GB zip-val = 0.0650 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:26:28.561946 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005229.sst): second pass time = 0.00's,118004652.095'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1188.697'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.503'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.411'MB/sec write SST data time = 0.02's, 4096.245'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 118004652.095'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63701 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 63701 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0648 GB dict = 0.00 MB all = 0.0661 GB } __ZipSize{ index = 0.0000 GB value = 0.0650 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index =3981.3125 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 15.686080 GB avg = 0.263 KB (by entry num) total key len = 1.530348 GB avg = 0.026 KB total ukey len = 1.053918 GB avg = 0.018 KB total ukey num = 0.059553727 Billion total entry num = 0.059553727 Billion write speed all = 36.506902107 MB/sec (with seq num) write speed all = 35.496647413 MB/sec (without seq num) 2024/11/19-15:26:28.573118 139851158521600 [compaction_job.cc:1948] [default] [JOB 231] Generated table #5229: 63701 keys, 64993098 bytes (need compaction), temperature: kUnknown 2024/11/19-15:26:28.573171 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001188573148, "cf_name": "default", "job": 231, "event": "table_file_creation", "file_number": 5229, "file_size": 64993098, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64991792, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1274020, "raw_average_key_size": 20, "raw_value_size": 64784554, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63701, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001188, "oldest_key_time": 0, "file_creation_time": 1732001188, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5229, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:28.588850 139851158521600 [compaction_job.cc:703] job-00231: subcompact[0], size: 0.131130 G, files: /storage/mytopling/.rocksdb [005228,005229] 2024/11/19-15:26:28.597249 139851158521600 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 12952261 2024/11/19-15:26:28.597387 139851158521600 (Original Log Time 2024/11/19-15:26:28.172925) EVENT_LOG_v1 {"time_micros": 1732001188172903, "job": 231, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5086, 5087, 5089, 5090], "score": 3.27737, "input_data_size": 134099264, "oldest_snapshot_seqno": 147924203} 2024/11/19-15:26:28.597394 139851158521600 (Original Log Time 2024/11/19-15:26:28.596915) [compaction_job.cc:2022] [default] [JOB 231] Compacted 4@1 files to L2 => 131130062 bytes 2024/11/19-15:26:28.597397 139851158521600 (Original Log Time 2024/11/19-15:26:28.597217) [compaction_job.cc:1224] [default] compacted to: files[6 49 94 58 0 0 0] max score 2.83 (125 files need compaction), MB/sec: 322.4 rd, 315.3 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.1 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128415, records dropped: 0 output_compression: Snappy 2024/11/19-15:26:28.597415 139851158521600 (Original Log Time 2024/11/19-15:26:28.597242) EVENT_LOG_v1 {"time_micros": 1732001188597229, "cf": "default", "job": 231, "event": "compaction_finished", "compaction_time_micros": 415944, "compaction_time_cpu_micros": 403158, "output_level": 2, "num_output_files": 2, "total_output_size": 131130062, "num_input_records": 128415, "num_output_records": 128415, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [6, 49, 94, 58, 0, 0, 0]} 2024/11/19-15:26:28.613180 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005053.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:28.613208 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001188613205, "job": 231, "event": "table_file_deletion", "file_number": 5053} 2024/11/19-15:26:28.633002 139851148035840 [dcompact_etcd.cc:1163] job-00241/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.277 sec, input = 158.093 MiB 2024/11/19-15:26:28.634775 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005055.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:28.634801 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001188634798, "job": 231, "event": "table_file_deletion", "file_number": 5055} 2024/11/19-15:26:28.658235 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005056.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:28.658257 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001188658254, "job": 231, "event": "table_file_deletion", "file_number": 5056} 2024/11/19-15:26:28.662274 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005057.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:28.662293 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001188662291, "job": 231, "event": "table_file_deletion", "file_number": 5057} 2024/11/19-15:26:28.666234 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005060.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:28.666253 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001188666251, "job": 231, "event": "table_file_deletion", "file_number": 5060} 2024/11/19-15:26:28.670174 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005061.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:28.670193 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001188670191, "job": 231, "event": "table_file_deletion", "file_number": 5061} 2024/11/19-15:26:28.674254 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005062.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:28.674273 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001188674271, "job": 231, "event": "table_file_deletion", "file_number": 5062} 2024/11/19-15:26:28.678523 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005064.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:28.678541 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001188678539, "job": 231, "event": "table_file_deletion", "file_number": 5064} 2024/11/19-15:26:28.678679 139851158521600 [compaction_job.cc:2437] [default] [JOB 242] Compacting 4@2 files to L3, score 2.83, subcompactions 1 : 1 2024/11/19-15:26:28.678688 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 240 Base level 2, inputs: [4788(63MB) 4789(61MB) 4790(63MB) 4791(61MB)] 2024/11/19-15:26:28.963685 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-01 2024/11/19-15:26:28.963778 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00239/att-01/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:26:28.968156 139851158521600 [dcompact_etcd.cc:1163] job-00242/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 253.782 MiB 2024/11/19-15:26:29.849171 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-01 2024/11/19-15:26:29.849253 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00238/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:26:30.236910 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-01 2024/11/19-15:26:30.237015 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00237/att-01/compact.done) = No such file or directory: 9.018 sec on http probe failed: 3, retry 2024/11/19-15:26:30.394215 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-02 2024/11/19-15:26:30.394289 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00233/att-02/compact.done) = No such file or directory: 7.008 sec on http probe failed: 2, retry 2024/11/19-15:26:30.973429 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-01 2024/11/19-15:26:30.973533 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00239/att-01/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:26:31.858579 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-01 2024/11/19-15:26:31.858656 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00238/att-01/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:26:32.246038 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-01 2024/11/19-15:26:32.246131 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00237/att-01/compact.done) = No such file or directory: 11.027 sec on http probe failed: 4, retry 2024/11/19-15:26:32.399317 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-02 2024/11/19-15:26:32.399386 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00233/att-02/compact.done) = No such file or directory: 9.016 sec on http probe failed: 3, retry 2024/11/19-15:26:32.591229 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-00 2024/11/19-15:26:32.591335 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00241/att-00/compact.done) = No such file or directory: 3.952 sec on http probe failed: 1, retry 2024/11/19-15:26:32.980294 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-01 2024/11/19-15:26:32.980379 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00239/att-01/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:26:33.398316 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-01 2024/11/19-15:26:33.398415 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00240/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:26:33.867695 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-01 2024/11/19-15:26:33.867793 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00238/att-01/compact.done) = No such file or directory: 9.018 sec on http probe failed: 3, retry 2024/11/19-15:26:33.973257 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-00 2024/11/19-15:26:33.973321 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00242/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:26:34.252170 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-01 2024/11/19-15:26:34.252242 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00237/att-01/compact.done) = No such file or directory: 13.036 sec on http probe failed: 5, giveup 2024/11/19-15:26:34.252260 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00237/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.640 sec, real 13.327 (submit 285.316 ms wait 13.036 s), input 255.640 MiB, send shutdown 2024/11/19-15:26:34.261089 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-01 2024/11/19-15:26:34.261294 139851169007360 [dcompact_etcd.cc:1773] job-00237/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:34.404566 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-02 2024/11/19-15:26:34.404663 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00233/att-02/compact.done) = No such file or directory: 11.022 sec on http probe failed: 4, retry 2024/11/19-15:26:34.542519 139851169007360 [dcompact_etcd.cc:1163] job-00237/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 255.640 MiB 2024/11/19-15:26:34.599767 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-00 2024/11/19-15:26:34.599832 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00241/att-00/compact.done) = No such file or directory: 5.958 sec on http probe failed: 2, retry 2024/11/19-15:26:34.985126 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-01 2024/11/19-15:26:34.985220 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00239/att-01/compact.done) = No such file or directory: 11.205 sec on http probe failed: 5, giveup 2024/11/19-15:26:34.985238 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00239/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.834 sec, real 11.515 (submit 305.281 ms wait 11.205 s), input 126.834 MiB, send shutdown 2024/11/19-15:26:34.994054 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-01 2024/11/19-15:26:34.994265 139851127064320 [dcompact_etcd.cc:1773] job-00239/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:35.257158 139851127064320 [dcompact_etcd.cc:1163] job-00239/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.263 sec, input = 126.834 MiB 2024/11/19-15:26:35.407371 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-01 2024/11/19-15:26:35.407460 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00240/att-01/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:26:35.877788 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-01 2024/11/19-15:26:35.877874 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00238/att-01/compact.done) = No such file or directory: 11.027 sec on http probe failed: 4, retry 2024/11/19-15:26:35.983322 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-00 2024/11/19-15:26:35.983393 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00242/att-00/compact.done) = No such file or directory: 7.005 sec on http probe failed: 2, retry 2024/11/19-15:26:36.409463 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-02 2024/11/19-15:26:36.409593 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00233/att-02/compact.done) = No such file or directory: 13.027 sec on http probe failed: 5, giveup 2024/11/19-15:26:36.409631 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00233/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 254.473 sec, real 13.330 (submit 298.324 ms wait 13.027 s), input 254.473 MiB, send shutdown 2024/11/19-15:26:36.414725 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00233/att-02 2024/11/19-15:26:36.415005 139851179493120 [dcompact_etcd.cc:1773] job-00233/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:36.416886 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00233) = success 2024/11/19-15:26:36.416903 139851179493120 [dcompact_etcd.cc:1773] job-00233/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:26:36.416942 139851179493120 (Original Log Time 2024/11/19-15:25:56.409877) EVENT_LOG_v1 {"time_micros": 1732001156409870, "job": 233, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [5174, 5175, 5183, 5184], "score": 3.22125, "input_data_size": 259243828, "oldest_snapshot_seqno": -1} 2024/11/19-15:26:36.416961 139851179493120 [compaction_job.cc:2437] [default] [JOB 233] Compacting 4@2 files to L3, score 3.22, subcompactions 1 : 1 2024/11/19-15:26:36.416974 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 231 Base level 2, inputs: [5174(62MB) 5175(61MB) 5183(62MB) 5184(61MB)] 2024/11/19-15:26:36.417257 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005230.sst) /storage/mytopling/..//tmp/Topling-82038-jXUvTC 2024/11/19-15:26:36.474949 139852349699840 [db_impl_write.cc:2181] [default] New memtable created with log file: #5231. Immutable memtables: 0. 2024/11/19-15:26:36.475310 139852349699840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 6 level-0 files rate 12563693 2024/11/19-15:26:36.475402 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 243] Syncing log #5218 2024/11/19-15:26:36.475502 139850957195008 (Original Log Time 2024/11/19-15:26:36.475388) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:26:36.475514 139850957195008 [flush_job.cc:879] [default] [JOB 243] Flushing memtable with next log file: 5231 2024/11/19-15:26:36.475535 139850957195008 EVENT_LOG_v1 {"time_micros": 1732001196475529, "job": 243, "event": "flush_started", "num_memtables": 1, "num_entries": 1450505, "num_deletes": 0, "total_data_size": 470533099, "memory_usage": 544472456, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:26:36.475539 139850957195008 [flush_job.cc:910] [default] [JOB 243] Level-0 flush table #5232: started 2024/11/19-15:26:36.475906 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000031.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/005232.sst in ConvertToSST, mem_size = 519.249 M, time = 358.081 us 2024/11/19-15:26:36.609066 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-00 2024/11/19-15:26:36.609180 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00241/att-00/compact.done) = No such file or directory: 7.967 sec on http probe failed: 3, retry 2024/11/19-15:26:36.640905 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005230.sst): first pass time = 0.22's, 600.201'MB/sec 2024/11/19-15:26:36.640910 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005230.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0018 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:26:36.640946 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005230.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.002 GB, dictZip workingMem = 0.0081 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:26:36.653904 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005230.sst): index pass time = 0.01's, 185.781'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 200604 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 2024/11/19-15:26:36.949770 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/005232.sst): fsize = 519.251 M, time(ms): open: 0.044, seek: 0.014, finish: 0.037, meta: 0.014, Flush: 0.013, sync: 473.715, close: 0.011, all: 473.848 2024/11/19-15:26:36.950947 139850957195008 [flush_job.cc:1083] [default] [JOB 243] Flush lasted 475445 microseconds, and 424991 cpu microseconds. 2024/11/19-15:26:36.972014 139850957195008 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 12186782 2024/11/19-15:26:36.972095 139850957195008 (Original Log Time 2024/11/19-15:26:36.949824) [flush_job.cc:1029] [default] [JOB 243] Level-0 flush table #5232: 544473882 bytes OK (needs compaction) 2024/11/19-15:26:36.972101 139850957195008 (Original Log Time 2024/11/19-15:26:36.950963) [memtable_list.cc:560] [default] Level-0 commit table #5232 started 2024/11/19-15:26:36.972104 139850957195008 (Original Log Time 2024/11/19-15:26:36.971983) [memtable_list.cc:759] [default] Level-0 commit table #5232: memtable #1 done 2024/11/19-15:26:36.972106 139850957195008 (Original Log Time 2024/11/19-15:26:36.972008) EVENT_LOG_v1 {"time_micros": 1732001196972002, "job": 243, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [7, 49, 94, 58, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:26:36.972109 139850957195008 (Original Log Time 2024/11/19-15:26:36.972068) [db_impl_compaction_flush.cc:310] [default] Level summary: files[7 49 94 58 0 0 0] max score 2.78 (122 files need compaction) 2024/11/19-15:26:37.422648 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-01 2024/11/19-15:26:37.422757 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00240/att-01/compact.done) = No such file or directory: 9.020 sec on http probe failed: 3, retry 2024/11/19-15:26:37.890981 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-01 2024/11/19-15:26:37.891079 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00238/att-01/compact.done) = No such file or directory: 13.038 sec on http probe failed: 5, giveup 2024/11/19-15:26:37.891106 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00238/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.642 sec, real 13.322 (submit 272.144 ms wait 13.038 s), input 255.642 MiB, send shutdown 2024/11/19-15:26:37.900089 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-01 2024/11/19-15:26:37.901038 139851137550080 [dcompact_etcd.cc:1773] job-00238/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:37.999166 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-00 2024/11/19-15:26:37.999271 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00242/att-00/compact.done) = No such file or directory: 9.018 sec on http probe failed: 3, retry 2024/11/19-15:26:38.206416 139851137550080 [dcompact_etcd.cc:1163] job-00238/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.305 sec, input = 255.642 MiB 2024/11/19-15:26:38.438040 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-02 2024/11/19-15:26:38.438144 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00239/att-02/compact.done) = No such file or directory: 3.172 sec on http probe failed: 1, retry 2024/11/19-15:26:38.626254 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-00 2024/11/19-15:26:38.626350 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00241/att-00/compact.done) = No such file or directory: 9.978 sec on http probe failed: 4, retry 2024/11/19-15:26:38.746119 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005230.sst): old prealloc_size = 147639500, real_size = 103779088 2024/11/19-15:26:38.746251 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005230.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 200604 prefix = 4 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 raw-val = 0.1294 GB zip-val = 0.1034 GB avg-val = 645.07 avg-zval = 515.39 2024/11/19-15:26:38.746311 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005230.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0005 GB, waited 2.105 sec, Key+Value = 0.134 GB 2024/11/19-15:26:38.946430 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005230.sst): second pass time = 2.10's, 61.484'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.20's, 524.602'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3064.138'MB/sec (index lex order gen) rebuild zvType time = 0.16's, 0.317'MB/sec write SST data time = 0.04's, 2504.251'MB/sec dict compress time = 0.00's, 2063.795'MB/sec z-dict build time = 0.00's, sample length = 1.348'MB, throughput = 22066.338'MB/sec zip my value time = 2.10's, unzip length = 0.129'GB zip my value throughput = 61.484'MB/sec zip pipeline throughput = 61.484'MB/sec entries = 200604 avg-key = 24.00 avg-zkey = 1.69 avg-val = 645.07 avg-zval = 515.39 usrkeys = 200604 avg-key = 16.00 avg-zkey = 1.69 avg-val = 645.07 avg-zval = 515.39 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0032 GB value = 0.1294 GB dict = 1.35 MB all = 0.1342 GB } __ZipSize{ index = 0.0003 GB value = 0.1034 GB dict = 1.35 MB all = 0.1051 GB } UnZip/Zip{ index = 9.4785 value = 1.2516 dict = 1.00 all = 1.2773 } Zip/UnZip{ index = 0.1055 value = 0.7990 dict = 1.00 all = 0.7829 } ---------------------------- total value len = 15.815483 GB avg = 0.265 KB (by entry num) total key len = 1.535162 GB avg = 0.026 KB total ukey len = 1.057128 GB avg = 0.018 KB total ukey num = 0.059754331 Billion total entry num = 0.059754331 Billion write speed all = 35.998813857 MB/sec (with seq num) write speed all = 35.006995988 MB/sec (without seq num) 2024/11/19-15:26:38.948892 139851179493120 [compaction_job.cc:1948] [default] [JOB 233] Generated table #5230: 200604 keys, 105080201 bytes, temperature: kUnknown 2024/11/19-15:26:38.948956 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001198948932, "cf_name": "default", "job": 233, "event": "table_file_creation", "file_number": 5230, "file_size": 105080201, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 103390144, "index_size": 338624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4814496, "raw_average_key_size": 24, "raw_value_size": 129403597, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 200604, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001196, "oldest_key_time": 0, "file_creation_time": 1732001196, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5230, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:38.976126 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005233.sst) /storage/mytopling/..//tmp/Topling-82038-3mUNmW 2024/11/19-15:26:39.104390 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005233.sst): first pass time = 0.13's,1034.080'MB/sec 2024/11/19-15:26:39.104391 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005233.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0018 GB, waited 0.000 sec, Key+Value = 0.133 GB 2024/11/19-15:26:39.104428 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005233.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.002 GB, dictZip workingMem = 0.0080 GB, waited 0.000 sec, Key+Value = 0.133 GB 2024/11/19-15:26:39.114720 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005233.sst): index pass time = 0.01's, 231.185'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 198239 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 2024/11/19-15:26:39.434004 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-01 2024/11/19-15:26:39.434126 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00240/att-01/compact.done) = No such file or directory: 11.034 sec on http probe failed: 4, retry 2024/11/19-15:26:39.558080 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-02 2024/11/19-15:26:39.558188 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00237/att-02/compact.done) = No such file or directory: 5.001 sec on http probe failed: 1, retry 2024/11/19-15:26:40.004473 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-00 2024/11/19-15:26:40.004593 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00242/att-00/compact.done) = No such file or directory: 11.031 sec on http probe failed: 4, retry 2024/11/19-15:26:40.448977 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-02 2024/11/19-15:26:40.449072 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00239/att-02/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:26:40.634950 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-00 2024/11/19-15:26:40.635048 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00241/att-00/compact.done) = No such file or directory: 11.993 sec on http probe failed: 5, giveup 2024/11/19-15:26:40.635073 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00241/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 158.093 sec, real 12.279 (submit 277.425 ms wait 11.993 s), input 158.093 MiB, send shutdown 2024/11/19-15:26:40.645614 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-00 2024/11/19-15:26:40.646543 139851148035840 [dcompact_etcd.cc:1773] job-00241/att-00: CleanFiles 0.001 sec : 0.000 2024/11/19-15:26:40.936704 139851148035840 [dcompact_etcd.cc:1163] job-00241/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.290 sec, input = 158.093 MiB 2024/11/19-15:26:41.051885 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005233.sst): old prealloc_size = 147639500, real_size = 102551440 2024/11/19-15:26:41.052018 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005233.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 198239 prefix = 4 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 raw-val = 0.1279 GB zip-val = 0.1022 GB avg-val = 644.97 avg-zval = 515.37 2024/11/19-15:26:41.052084 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005233.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 1.948 sec, Key+Value = 0.133 GB 2024/11/19-15:26:41.249180 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005233.sst): second pass time = 1.95's, 65.668'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.20's, 526.359'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3224.259'MB/sec (index lex order gen) rebuild zvType time = 0.16's, 0.319'MB/sec write SST data time = 0.04's, 2481.560'MB/sec dict compress time = 0.00's, 2019.999'MB/sec z-dict build time = 0.00's, sample length = 1.337'MB, throughput = 17933.273'MB/sec zip my value time = 1.95's, unzip length = 0.128'GB zip my value throughput = 65.668'MB/sec zip pipeline throughput = 65.668'MB/sec entries = 198239 avg-key = 24.00 avg-zkey = 1.69 avg-val = 644.97 avg-zval = 515.37 usrkeys = 198239 avg-key = 16.00 avg-zkey = 1.69 avg-val = 644.97 avg-zval = 515.37 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0032 GB value = 0.1279 GB dict = 1.34 MB all = 0.1326 GB } __ZipSize{ index = 0.0003 GB value = 0.1022 GB dict = 1.34 MB all = 0.1038 GB } UnZip/Zip{ index = 9.4761 value = 1.2515 dict = 1.00 all = 1.2771 } Zip/UnZip{ index = 0.1055 value = 0.7991 dict = 1.00 all = 0.7830 } ---------------------------- total value len = 15.943342 GB avg = 0.266 KB (by entry num) total key len = 1.539920 GB avg = 0.026 KB total ukey len = 1.060299 GB avg = 0.018 KB total ukey num = 0.059952570 Billion total entry num = 0.059952570 Billion write speed all = 36.101480841 MB/sec (with seq num) write speed all = 35.111104278 MB/sec (without seq num) 2024/11/19-15:26:41.251551 139851179493120 [compaction_job.cc:1948] [default] [JOB 233] Generated table #5233: 198239 keys, 103841907 bytes, temperature: kUnknown 2024/11/19-15:26:41.251602 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001201251578, "cf_name": "default", "job": 233, "event": "table_file_creation", "file_number": 5233, "file_size": 103841907, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 102166976, "index_size": 334720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4757736, "raw_average_key_size": 24, "raw_value_size": 127858233, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 198239, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001198, "oldest_key_time": 0, "file_creation_time": 1732001198, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5233, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:41.278545 139851179493120 [compaction_job.cc:703] job-00233: subcompact[0], size: 0.208922 G, files: /storage/mytopling/.rocksdb [005230,005233] 2024/11/19-15:26:41.280388 139851179493120 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 12563692 2024/11/19-15:26:41.280546 139851179493120 (Original Log Time 2024/11/19-15:26:36.417014) EVENT_LOG_v1 {"time_micros": 1732001196416990, "job": 233, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [5174, 5175, 5183, 5184], "score": 3.22125, "input_data_size": 259243828, "oldest_snapshot_seqno": -1} 2024/11/19-15:26:41.280553 139851179493120 (Original Log Time 2024/11/19-15:26:41.280054) [compaction_job.cc:2022] [default] [JOB 233] Compacted 4@2 files to L3 => 208922108 bytes 2024/11/19-15:26:41.280556 139851179493120 (Original Log Time 2024/11/19-15:26:41.280355) [compaction_job.cc:1224] [default] compacted to: files[7 49 90 60 0 0 0] max score 2.78 (122 files need compaction), MB/sec: 53.3 rd, 43.0 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(247.2, 0.0 +0.0 blob) out(199.2 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 398843, records dropped: 0 output_compression: Snappy 2024/11/19-15:26:41.280574 139851179493120 (Original Log Time 2024/11/19-15:26:41.280381) EVENT_LOG_v1 {"time_micros": 1732001201280368, "cf": "default", "job": 233, "event": "compaction_finished", "compaction_time_micros": 4861547, "compaction_time_cpu_micros": 1645561, "output_level": 3, "num_output_files": 2, "total_output_size": 208922108, "num_input_records": 398843, "num_output_records": 398843, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [7, 49, 90, 60, 0, 0, 0]} 2024/11/19-15:26:41.285041 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005065.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:41.285069 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001201285066, "job": 233, "event": "table_file_deletion", "file_number": 5065} 2024/11/19-15:26:41.289436 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005066.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:41.289457 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001201289455, "job": 233, "event": "table_file_deletion", "file_number": 5066} 2024/11/19-15:26:41.293562 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005068.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:41.293583 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001201293581, "job": 233, "event": "table_file_deletion", "file_number": 5068} 2024/11/19-15:26:41.297854 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005069.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:41.297874 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001201297871, "job": 233, "event": "table_file_deletion", "file_number": 5069} 2024/11/19-15:26:41.301871 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005070.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:41.301889 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001201301887, "job": 233, "event": "table_file_deletion", "file_number": 5070} 2024/11/19-15:26:41.305938 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005071.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:41.305957 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001201305955, "job": 233, "event": "table_file_deletion", "file_number": 5071} 2024/11/19-15:26:41.310407 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005073.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:41.310425 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001201310423, "job": 233, "event": "table_file_deletion", "file_number": 5073} 2024/11/19-15:26:41.315096 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005074.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:41.315115 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001201315113, "job": 233, "event": "table_file_deletion", "file_number": 5074} 2024/11/19-15:26:41.315253 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005169.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:41.315262 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001201315261, "job": 233, "event": "table_file_deletion", "file_number": 5169} 2024/11/19-15:26:41.315395 139851179493120 [compaction_job.cc:2437] [default] [JOB 244] Compacting 4@1 files to L2, score 2.78, subcompactions 1 : 1 2024/11/19-15:26:41.315404 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 242 Base level 1, inputs: [5097(31MB) 5099(31MB) 5101(31MB) 5103(31MB)] 2024/11/19-15:26:41.441322 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-01 2024/11/19-15:26:41.441396 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00240/att-01/compact.done) = No such file or directory: 13.046 sec on http probe failed: 5, giveup 2024/11/19-15:26:41.441414 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00240/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 219.968 sec, real 13.338 (submit 285.237 ms wait 13.046 s), input 219.968 MiB, send shutdown 2024/11/19-15:26:41.449851 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-01 2024/11/19-15:26:41.450064 139850978166528 [dcompact_etcd.cc:1773] job-00240/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:41.566607 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-02 2024/11/19-15:26:41.566673 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00237/att-02/compact.done) = No such file or directory: 7.016 sec on http probe failed: 2, retry 2024/11/19-15:26:41.643203 139851179493120 [dcompact_etcd.cc:1163] job-00244/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.327 sec, input = 126.952 MiB 2024/11/19-15:26:41.753456 139850978166528 [dcompact_etcd.cc:1163] job-00240/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.303 sec, input = 219.968 MiB 2024/11/19-15:26:42.013786 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-00 2024/11/19-15:26:42.013864 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00242/att-00/compact.done) = No such file or directory: 13.037 sec on http probe failed: 5, giveup 2024/11/19-15:26:42.013882 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00242/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.782 sec, real 13.335 (submit 289.011 ms wait 13.037 s), input 253.782 MiB, send shutdown 2024/11/19-15:26:42.023522 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-00 2024/11/19-15:26:42.023754 139851158521600 [dcompact_etcd.cc:1773] job-00242/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:42.326545 139851158521600 [dcompact_etcd.cc:1163] job-00242/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.303 sec, input = 253.782 MiB 2024/11/19-15:26:42.459864 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-02 2024/11/19-15:26:42.459950 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00239/att-02/compact.done) = No such file or directory: 7.192 sec on http probe failed: 3, retry 2024/11/19-15:26:43.215965 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-02 2024/11/19-15:26:43.216052 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00238/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:26:43.575353 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-02 2024/11/19-15:26:43.575454 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00237/att-02/compact.done) = No such file or directory: 9.024 sec on http probe failed: 3, retry 2024/11/19-15:26:44.468682 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-02 2024/11/19-15:26:44.468764 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00239/att-02/compact.done) = No such file or directory: 9.203 sec on http probe failed: 4, retry 2024/11/19-15:26:44.826531 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-00 2024/11/19-15:26:44.826630 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00244/att-00/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:26:44.897838 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-01 2024/11/19-15:26:44.897914 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00241/att-01/compact.done) = No such file or directory: 3.952 sec on http probe failed: 1, retry 2024/11/19-15:26:45.225653 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-02 2024/11/19-15:26:45.225722 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00238/att-02/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:26:45.580433 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-02 2024/11/19-15:26:45.580502 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00237/att-02/compact.done) = No such file or directory: 11.033 sec on http probe failed: 4, retry 2024/11/19-15:26:46.478123 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-02 2024/11/19-15:26:46.478221 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00239/att-02/compact.done) = No such file or directory: 11.212 sec on http probe failed: 5, giveup 2024/11/19-15:26:46.478240 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00239/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.834 sec, real 11.484 (submit 262.820 ms wait 11.212 s), input 126.834 MiB, send shutdown 2024/11/19-15:26:46.487159 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00239/att-02 2024/11/19-15:26:46.487378 139851127064320 [dcompact_etcd.cc:1773] job-00239/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:46.489233 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00239) = success 2024/11/19-15:26:46.489241 139851127064320 [dcompact_etcd.cc:1773] job-00239/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:26:46.489260 139851127064320 (Original Log Time 2024/11/19-15:26:11.952873) EVENT_LOG_v1 {"time_micros": 1732001171952864, "job": 239, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5092, 5093, 5094, 5096], "score": 3.02759, "input_data_size": 134099840, "oldest_snapshot_seqno": 148695933} 2024/11/19-15:26:46.489268 139851127064320 [compaction_job.cc:2437] [default] [JOB 239] Compacting 4@1 files to L2, score 3.03, subcompactions 1 : 1 2024/11/19-15:26:46.489276 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 237 Base level 1, inputs: [5092(31MB) 5093(31MB) 5094(31MB) 5096(31MB)] 2024/11/19-15:26:46.489511 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005234.sst) /storage/mytopling/..//tmp/Topling-82038-9MKZ14 2024/11/19-15:26:46.543363 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005234.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:26:46.546707 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005234.sst): index pass time = 0.00's, 156.455'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64714 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:26:46.610230 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005234.sst): first pass time = 0.12's, 556.017'MB/sec 2024/11/19-15:26:46.610284 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005234.sst): old prealloc_size = 73819750, real_size = 66152464 2024/11/19-15:26:46.610341 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005234.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64714 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.02 avg-zval =1020.27 2024/11/19-15:26:46.610408 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005234.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.067 GB 2024/11/19-15:26:46.670183 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005234.sst): second pass time = 0.00's,232563204.947'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1104.455'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2044.486'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.379'MB/sec write SST data time = 0.02's, 3861.627'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 232563204.947'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64714 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.02 avg-zval = 1020.27 usrkeys = 64714 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.02 avg-zval = 1020.27 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8063 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0464 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 16.009157 GB avg = 0.267 KB (by entry num) total key len = 1.541214 GB avg = 0.026 KB total ukey len = 1.061076 GB avg = 0.018 KB total ukey num = 0.060017284 Billion total entry num = 0.060017284 Billion write speed all = 35.838879111 MB/sec (with seq num) write speed all = 34.858408824 MB/sec (without seq num) 2024/11/19-15:26:46.673840 139851127064320 [compaction_job.cc:1948] [default] [JOB 239] Generated table #5234: 64714 keys, 66137668 bytes (need compaction), temperature: kUnknown 2024/11/19-15:26:46.673893 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001206673870, "cf_name": "default", "job": 239, "event": "table_file_creation", "file_number": 5234, "file_size": 66137668, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025920, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294280, "raw_average_key_size": 20, "raw_value_size": 65815387, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64714, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001206, "oldest_key_time": 0, "file_creation_time": 1732001206, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5234, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:46.689106 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005235.sst) /storage/mytopling/..//tmp/Topling-82038-pNdPF0 2024/11/19-15:26:46.740571 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005235.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:26:46.743853 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005235.sst): index pass time = 0.00's, 156.320'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63540 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:26:46.762666 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-02 2024/11/19-15:26:46.762743 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00240/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:26:46.806264 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005235.sst): first pass time = 0.12's, 562.428'MB/sec 2024/11/19-15:26:46.806309 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005235.sst): old prealloc_size = 73819750, real_size = 64945344 2024/11/19-15:26:46.806367 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005235.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63540 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0646 GB zip-val = 0.0648 GB avg-val =1016.91 avg-zval =1020.16 2024/11/19-15:26:46.806430 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005235.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.066 GB 2024/11/19-15:26:46.835156 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-00 2024/11/19-15:26:46.835253 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00244/att-00/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:26:46.864644 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005235.sst): second pass time = 0.00's,215380810.000'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1113.369'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2029.081'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.381'MB/sec write SST data time = 0.02's, 3929.460'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 215380810.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63540 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.91 avg-zval = 1020.16 usrkeys = 63540 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.91 avg-zval = 1020.16 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7917 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0648 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index = 7.0412 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 16.073771 GB avg = 0.268 KB (by entry num) total key len = 1.542485 GB avg = 0.026 KB total ukey len = 1.061838 GB avg = 0.018 KB total ukey num = 0.060080824 Billion total entry num = 0.060080824 Billion write speed all = 35.959140643 MB/sec (with seq num) write speed all = 34.978021944 MB/sec (without seq num) 2024/11/19-15:26:46.868598 139851127064320 [compaction_job.cc:1948] [default] [JOB 239] Generated table #5235: 63540 keys, 64930820 bytes (need compaction), temperature: kUnknown 2024/11/19-15:26:46.868651 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001206868628, "cf_name": "default", "job": 239, "event": "table_file_creation", "file_number": 5235, "file_size": 64930820, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64820976, "index_size": 108288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1270800, "raw_average_key_size": 20, "raw_value_size": 64614243, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63540, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001206, "oldest_key_time": 0, "file_creation_time": 1732001206, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5235, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:46.883522 139851127064320 [compaction_job.cc:703] job-00239: subcompact[0], size: 0.131068 G, files: /storage/mytopling/.rocksdb [005234,005235] 2024/11/19-15:26:46.884255 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 12952259 2024/11/19-15:26:46.884381 139851127064320 (Original Log Time 2024/11/19-15:26:46.489303) EVENT_LOG_v1 {"time_micros": 1732001206489287, "job": 239, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5092, 5093, 5094, 5096], "score": 3.02759, "input_data_size": 134099840, "oldest_snapshot_seqno": 148695933} 2024/11/19-15:26:46.884387 139851127064320 (Original Log Time 2024/11/19-15:26:46.883939) [compaction_job.cc:2022] [default] [JOB 239] Compacted 4@1 files to L2 => 131068488 bytes 2024/11/19-15:26:46.884390 139851127064320 (Original Log Time 2024/11/19-15:26:46.884226) [compaction_job.cc:1224] [default] compacted to: files[7 45 92 60 0 0 0] max score 2.75 (120 files need compaction), MB/sec: 340.2 rd, 332.5 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128254, records dropped: 0 output_compression: Snappy 2024/11/19-15:26:46.884406 139851127064320 (Original Log Time 2024/11/19-15:26:46.884249) EVENT_LOG_v1 {"time_micros": 1732001206884237, "cf": "default", "job": 239, "event": "compaction_finished", "compaction_time_micros": 394229, "compaction_time_cpu_micros": 392404, "output_level": 2, "num_output_files": 2, "total_output_size": 131068488, "num_input_records": 128254, "num_output_records": 128254, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [7, 45, 92, 60, 0, 0, 0]} 2024/11/19-15:26:46.884478 139851127064320 [compaction_job.cc:2437] [default] [JOB 245] Compacting 4@2 files to L3, score 2.75, subcompactions 1 : 1 2024/11/19-15:26:46.884487 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 243 Base level 2, inputs: [4631(31MB) 4617(31MB) 4787(62MB) 4619(31MB)] 2024/11/19-15:26:46.907406 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-01 2024/11/19-15:26:46.907463 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00241/att-01/compact.done) = No such file or directory: 5.961 sec on http probe failed: 2, retry 2024/11/19-15:26:47.198341 139851127064320 [dcompact_etcd.cc:1163] job-00245/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.313 sec, input = 158.601 MiB 2024/11/19-15:26:47.235371 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-02 2024/11/19-15:26:47.235430 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00238/att-02/compact.done) = No such file or directory: 9.019 sec on http probe failed: 3, retry 2024/11/19-15:26:47.334709 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-01 2024/11/19-15:26:47.334773 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00242/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:26:47.589265 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-02 2024/11/19-15:26:47.589329 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00237/att-02/compact.done) = No such file or directory: 13.038 sec on http probe failed: 5, giveup 2024/11/19-15:26:47.589345 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00237/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.640 sec, real 13.328 (submit 281.154 ms wait 13.038 s), input 255.640 MiB, send shutdown 2024/11/19-15:26:47.598395 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00237/att-02 2024/11/19-15:26:47.598611 139851169007360 [dcompact_etcd.cc:1773] job-00237/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:47.600605 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00237) = success 2024/11/19-15:26:47.600612 139851169007360 [dcompact_etcd.cc:1773] job-00237/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:26:47.600630 139851169007360 (Original Log Time 2024/11/19-15:26:07.563389) EVENT_LOG_v1 {"time_micros": 1732001167563382, "job": 237, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [5185, 5186, 5189, 5190], "score": 3.16215, "input_data_size": 221723616, "oldest_snapshot_seqno": 148605887} 2024/11/19-15:26:47.600637 139851169007360 [compaction_job.cc:2437] [default] [JOB 237] Compacting 4@2 files to L3, score 3.16, subcompactions 1 : 1 2024/11/19-15:26:47.600645 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 235 Base level 2, inputs: [5185(52MB) 5186(52MB) 5189(52MB) 5190(52MB)] 2024/11/19-15:26:47.600916 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005236.sst) /storage/mytopling/..//tmp/Topling-82038-EBS6se 2024/11/19-15:26:47.925545 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005236.sst): first pass time = 0.32's, 413.482'MB/sec 2024/11/19-15:26:47.925551 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005236.sst: sumWaitingMem = 0.012 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0121 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:26:47.925585 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005236.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.012 GB, dictZip workingMem = 0.0065 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:26:47.995411 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005236.sst): index pass time = 0.07's, 212.413'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 926952 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:26:48.781178 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-02 2024/11/19-15:26:48.781270 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00240/att-02/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:26:48.843059 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-00 2024/11/19-15:26:48.843141 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00244/att-00/compact.done) = No such file or directory: 7.194 sec on http probe failed: 3, retry 2024/11/19-15:26:48.926665 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-01 2024/11/19-15:26:48.926754 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00241/att-01/compact.done) = No such file or directory: 7.975 sec on http probe failed: 3, retry 2024/11/19-15:26:49.242873 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-02 2024/11/19-15:26:49.242960 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00238/att-02/compact.done) = No such file or directory: 11.029 sec on http probe failed: 4, retry 2024/11/19-15:26:49.340105 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-01 2024/11/19-15:26:49.340169 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00242/att-01/compact.done) = No such file or directory: 7.008 sec on http probe failed: 2, retry 2024/11/19-15:26:49.385090 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005236.sst): old prealloc_size = 147639500, real_size = 36366320 2024/11/19-15:26:49.385527 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005236.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926952 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.79 avg-zval = 37.09 2024/11/19-15:26:49.385585 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005236.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 1.460 sec, Key+Value = 0.134 GB 2024/11/19-15:26:49.914676 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005236.sst): second pass time = 1.46's, 74.180'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.53's, 70.292'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 4127.510'MB/sec (index lex order gen) rebuild zvType time = 0.43's, 0.534'MB/sec write SST data time = 0.09's, 392.102'MB/sec dict compress time = 0.00's, 1993.553'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 20678.917'MB/sec zip my value time = 1.46's, unzip length = 0.108'GB zip my value throughput = 74.180'MB/sec zip pipeline throughput = 74.180'MB/sec entries = 926952 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.79 avg-zval = 37.09 usrkeys = 926952 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.79 avg-zval = 37.09 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5801 value = 3.1488 dict = 1.00 all = 3.6056 } Zip/UnZip{ index = 0.0945 value = 0.3176 dict = 1.00 all = 0.2773 } ---------------------------- total value len = 16.182034 GB avg = 0.265 KB (by entry num) total key len = 1.568440 GB avg = 0.026 KB total ukey len = 1.080377 GB avg = 0.018 KB total ukey num = 0.061007776 Billion total entry num = 0.061007776 Billion write speed all = 36.008925938 MB/sec (with seq num) write speed all = 35.018834303 MB/sec (without seq num) 2024/11/19-15:26:49.915718 139851169007360 [compaction_job.cc:1948] [default] [JOB 237] Generated table #5236: 926952 keys, 37224435 bytes, temperature: kUnknown 2024/11/19-15:26:49.915770 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001209915746, "cf_name": "default", "job": 237, "event": "table_file_creation", "file_number": 5236, "file_size": 37224435, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34382112, "index_size": 1752256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954656, "raw_average_key_size": 28, "raw_value_size": 108263200, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926952, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001207, "oldest_key_time": 0, "file_creation_time": 1732001207, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5236, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:49.936484 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005237.sst) /storage/mytopling/..//tmp/Topling-82038-rHaZO0 2024/11/19-15:26:50.257144 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005237.sst): first pass time = 0.32's, 417.415'MB/sec 2024/11/19-15:26:50.257147 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005237.sst: sumWaitingMem = 0.012 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0120 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:26:50.257182 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005237.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.012 GB, dictZip workingMem = 0.0065 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:26:50.327299 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005237.sst): index pass time = 0.07's, 210.948'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 924368 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:26:50.791363 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-02 2024/11/19-15:26:50.791445 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00240/att-02/compact.done) = No such file or directory: 9.028 sec on http probe failed: 3, retry 2024/11/19-15:26:50.851418 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-00 2024/11/19-15:26:50.851526 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00244/att-00/compact.done) = No such file or directory: 9.200 sec on http probe failed: 4, retry 2024/11/19-15:26:50.937095 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-01 2024/11/19-15:26:50.937187 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00241/att-01/compact.done) = No such file or directory: 9.990 sec on http probe failed: 4, retry 2024/11/19-15:26:51.175021 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-00 2024/11/19-15:26:51.175148 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00245/att-00/compact.done) = No such file or directory: 3.967 sec on http probe failed: 1, retry 2024/11/19-15:26:51.254976 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-02 2024/11/19-15:26:51.255093 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00238/att-02/compact.done) = No such file or directory: 13.037 sec on http probe failed: 5, giveup 2024/11/19-15:26:51.255119 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00238/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 255.642 sec, real 13.354 (submit 305.271 ms wait 13.037 s), input 255.642 MiB, send shutdown 2024/11/19-15:26:51.264577 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00238/att-02 2024/11/19-15:26:51.279289 139851137550080 [dcompact_etcd.cc:1773] job-00238/att-02: CleanFiles 0.015 sec : 0.000 2024/11/19-15:26:51.337814 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00238) = success 2024/11/19-15:26:51.337834 139851137550080 [dcompact_etcd.cc:1773] job-00238/att-03: CleanFiles 0.058 sec : 0.000 2024/11/19-15:26:51.337864 139851137550080 (Original Log Time 2024/11/19-15:26:11.154945) EVENT_LOG_v1 {"time_micros": 1732001171154938, "job": 238, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [5193, 5194, 5197, 5199], "score": 3.07839, "input_data_size": 221718240, "oldest_snapshot_seqno": 148685929} 2024/11/19-15:26:51.337876 139851137550080 [compaction_job.cc:2437] [default] [JOB 238] Compacting 4@2 files to L3, score 3.08, subcompactions 1 : 1 2024/11/19-15:26:51.337888 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 236 Base level 2, inputs: [5193(52MB) 5194(52MB) 5197(52MB) 5199(52MB)] 2024/11/19-15:26:51.338246 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005238.sst) /storage/mytopling/..//tmp/Topling-82038-wOdWk1 2024/11/19-15:26:51.354682 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-01 2024/11/19-15:26:51.354762 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00242/att-01/compact.done) = No such file or directory: 9.014 sec on http probe failed: 3, retry 2024/11/19-15:26:51.727123 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005238.sst): first pass time = 0.39's, 345.159'MB/sec 2024/11/19-15:26:51.727125 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005238.sst: sumWaitingMem = 0.012 GB, sumWorkingMem = 0.006 GB, nltTrie workingMem = 0.0121 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:26:51.727167 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005238.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.019 GB, dictZip workingMem = 0.0065 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:26:51.753530 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005237.sst): old prealloc_size = 147639500, real_size = 36292448 2024/11/19-15:26:51.754021 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005237.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924368 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1080 GB zip-val = 0.0343 GB avg-val = 116.79 avg-zval = 37.11 2024/11/19-15:26:51.754115 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005237.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.019 GB, reorder workingMem = 0.0023 GB, waited 1.497 sec, Key+Value = 0.134 GB 2024/11/19-15:26:51.816310 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005238.sst): index pass time = 0.09's, 166.375'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 926951 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:26:52.293352 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005237.sst): second pass time = 1.50's, 72.150'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.54's, 68.818'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3750.654'MB/sec (index lex order gen) rebuild zvType time = 0.43's, 0.533'MB/sec write SST data time = 0.11's, 351.162'MB/sec dict compress time = 0.00's, 1860.306'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 16957.340'MB/sec zip my value time = 1.50's, unzip length = 0.108'GB zip my value throughput = 72.150'MB/sec zip pipeline throughput = 72.150'MB/sec entries = 924368 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 37.11 usrkeys = 924368 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 37.11 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1080 GB dict = 1.08 MB all = 0.1338 GB } __ZipSize{ index = 0.0018 GB value = 0.0343 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5130 value = 3.1472 dict = 1.00 all = 3.6028 } Zip/UnZip{ index = 0.0951 value = 0.3177 dict = 1.00 all = 0.2776 } ---------------------------- total value len = 16.289993 GB avg = 0.263 KB (by entry num) total key len = 1.594322 GB avg = 0.026 KB total ukey len = 1.098865 GB avg = 0.018 KB total ukey num = 0.061932144 Billion total entry num = 0.061932144 Billion write speed all = 36.106209471 MB/sec (with seq num) write speed all = 35.105943029 MB/sec (without seq num) 2024/11/19-15:26:52.294453 139851169007360 [compaction_job.cc:1948] [default] [JOB 237] Generated table #5237: 924368 keys, 37148861 bytes, temperature: kUnknown 2024/11/19-15:26:52.294506 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001212294482, "cf_name": "default", "job": 237, "event": "table_file_creation", "file_number": 5237, "file_size": 37148861, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34302672, "index_size": 1758528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25882304, "raw_average_key_size": 28, "raw_value_size": 107958184, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 924368, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001209, "oldest_key_time": 0, "file_creation_time": 1732001209, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5237, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:52.314862 139851169007360 [compaction_job.cc:703] job-00237: subcompact[0], size: 0.074373 G, files: /storage/mytopling/.rocksdb [005236,005237] 2024/11/19-15:26:52.317259 139851169007360 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 13352844 2024/11/19-15:26:52.317424 139851169007360 (Original Log Time 2024/11/19-15:26:47.600673) EVENT_LOG_v1 {"time_micros": 1732001207600658, "job": 237, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [5185, 5186, 5189, 5190], "score": 3.16215, "input_data_size": 221723616, "oldest_snapshot_seqno": 148605887} 2024/11/19-15:26:52.317434 139851169007360 (Original Log Time 2024/11/19-15:26:52.316794) [compaction_job.cc:2022] [default] [JOB 237] Compacted 4@2 files to L3 => 74373296 bytes 2024/11/19-15:26:52.317439 139851169007360 (Original Log Time 2024/11/19-15:26:52.317228) [compaction_job.cc:1224] [default] compacted to: files[7 45 88 62 0 0 0] max score 2.65 (116 files need compaction), MB/sec: 47.0 rd, 15.8 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(211.5, 0.0 +0.0 blob) out(70.9 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 1851320, records dropped: 0 output_compression: Snappy 2024/11/19-15:26:52.317463 139851169007360 (Original Log Time 2024/11/19-15:26:52.317252) EVENT_LOG_v1 {"time_micros": 1732001212317240, "cf": "default", "job": 237, "event": "compaction_finished", "compaction_time_micros": 4714202, "compaction_time_cpu_micros": 3832009, "output_level": 3, "num_output_files": 2, "total_output_size": 74373296, "num_input_records": 1851320, "num_output_records": 1851320, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [7, 45, 88, 62, 0, 0, 0]} 2024/11/19-15:26:52.321761 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005075.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:52.321784 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001212321782, "job": 237, "event": "table_file_deletion", "file_number": 5075} 2024/11/19-15:26:52.325853 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005077.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:52.325873 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001212325871, "job": 237, "event": "table_file_deletion", "file_number": 5077} 2024/11/19-15:26:52.329768 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005078.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:52.329786 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001212329784, "job": 237, "event": "table_file_deletion", "file_number": 5078} 2024/11/19-15:26:52.333934 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005079.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:52.333953 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001212333951, "job": 237, "event": "table_file_deletion", "file_number": 5079} 2024/11/19-15:26:52.334069 139851169007360 [compaction_job.cc:2437] [default] [JOB 246] Compacting 4@2 files to L3, score 2.65, subcompactions 1 : 1 2024/11/19-15:26:52.334078 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 244 Base level 2, inputs: [4630(31MB) 4792(62MB) 4793(62MB) 4796(62MB)] 2024/11/19-15:26:52.626652 139851169007360 [dcompact_etcd.cc:1163] job-00246/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.292 sec, input = 222.689 MiB 2024/11/19-15:26:52.803416 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-02 2024/11/19-15:26:52.803535 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00240/att-02/compact.done) = No such file or directory: 11.038 sec on http probe failed: 4, retry 2024/11/19-15:26:52.863466 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-00 2024/11/19-15:26:52.863558 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00244/att-00/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:26:52.863584 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00244/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.952 sec, real 11.548 (submit 327.363 ms wait 11.208 s), input 126.952 MiB, send shutdown 2024/11/19-15:26:52.875601 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-00 2024/11/19-15:26:52.875869 139851179493120 [dcompact_etcd.cc:1773] job-00244/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:52.942333 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-01 2024/11/19-15:26:52.942404 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00241/att-01/compact.done) = No such file or directory: 12.001 sec on http probe failed: 5, giveup 2024/11/19-15:26:52.942422 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00241/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 158.093 sec, real 12.296 (submit 290.035 ms wait 12.001 s), input 158.093 MiB, send shutdown 2024/11/19-15:26:52.951722 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-01 2024/11/19-15:26:52.951935 139851148035840 [dcompact_etcd.cc:1773] job-00241/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:53.152627 139851179493120 [dcompact_etcd.cc:1163] job-00244/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.277 sec, input = 126.952 MiB 2024/11/19-15:26:53.180869 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-00 2024/11/19-15:26:53.180944 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00245/att-00/compact.done) = No such file or directory: 5.977 sec on http probe failed: 2, retry 2024/11/19-15:26:53.249291 139851148035840 [dcompact_etcd.cc:1163] job-00241/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.297 sec, input = 158.093 MiB 2024/11/19-15:26:53.255716 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005238.sst): old prealloc_size = 147639500, real_size = 36381872 2024/11/19-15:26:53.256163 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005238.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926951 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.79 avg-zval = 37.10 2024/11/19-15:26:53.256219 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005238.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 1.529 sec, Key+Value = 0.134 GB 2024/11/19-15:26:53.364828 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-01 2024/11/19-15:26:53.364895 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00242/att-01/compact.done) = No such file or directory: 11.028 sec on http probe failed: 4, retry 2024/11/19-15:26:53.794811 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005238.sst): second pass time = 1.53's, 70.828'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.54's, 69.080'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 4054.394'MB/sec (index lex order gen) rebuild zvType time = 0.44's, 0.524'MB/sec write SST data time = 0.10's, 387.168'MB/sec dict compress time = 0.00's, 1833.400'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 21917.556'MB/sec zip my value time = 1.53's, unzip length = 0.108'GB zip my value throughput = 70.828'MB/sec zip pipeline throughput = 70.828'MB/sec entries = 926951 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 37.10 usrkeys = 926951 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 37.10 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 10 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5106 value = 3.1485 dict = 1.00 all = 3.6042 } Zip/UnZip{ index = 0.0951 value = 0.3176 dict = 1.00 all = 0.2775 } ---------------------------- total value len = 16.398256 GB avg = 0.261 KB (by entry num) total key len = 1.620277 GB avg = 0.026 KB total ukey len = 1.117404 GB avg = 0.018 KB total ukey num = 0.062859095 Billion total entry num = 0.062859095 Billion write speed all = 36.267242824 MB/sec (with seq num) write speed all = 35.255073344 MB/sec (without seq num) 2024/11/19-15:26:53.796186 139851137550080 [compaction_job.cc:1948] [default] [JOB 238] Generated table #5238: 926951 keys, 37239431 bytes, temperature: kUnknown 2024/11/19-15:26:53.796237 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001213796214, "cf_name": "default", "job": 238, "event": "table_file_creation", "file_number": 5238, "file_size": 37239431, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34386080, "index_size": 1763840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954628, "raw_average_key_size": 28, "raw_value_size": 108263126, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926951, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001211, "oldest_key_time": 0, "file_creation_time": 1732001211, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5238, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:53.818434 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005239.sst) /storage/mytopling/..//tmp/Topling-82038-KTVhtT 2024/11/19-15:26:54.145111 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005239.sst): first pass time = 0.33's, 409.731'MB/sec 2024/11/19-15:26:54.145115 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005239.sst: sumWaitingMem = 0.012 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0120 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:26:54.145139 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005239.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.012 GB, dictZip workingMem = 0.0065 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:26:54.215388 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005239.sst): index pass time = 0.07's, 210.535'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 924301 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:26:54.821554 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-02 2024/11/19-15:26:54.821667 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00240/att-02/compact.done) = No such file or directory: 13.051 sec on http probe failed: 5, giveup 2024/11/19-15:26:54.821695 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00240/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 219.968 sec, real 13.372 (submit 303.317 ms wait 13.051 s), input 219.968 MiB, send shutdown 2024/11/19-15:26:54.837587 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00240/att-02 2024/11/19-15:26:54.837845 139850978166528 [dcompact_etcd.cc:1773] job-00240/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:54.838143 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00240) = success 2024/11/19-15:26:54.838154 139850978166528 [dcompact_etcd.cc:1773] job-00240/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:54.838177 139850978166528 (Original Log Time 2024/11/19-15:26:14.757339) EVENT_LOG_v1 {"time_micros": 1732001174757330, "job": 240, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [5198, 5200, 5195, 5196], "score": 2.91196, "input_data_size": 190769312, "oldest_snapshot_seqno": 148755002} 2024/11/19-15:26:54.838188 139850978166528 [compaction_job.cc:2437] [default] [JOB 240] Compacting 4@2 files to L3, score 2.91, subcompactions 1 : 1 2024/11/19-15:26:54.838198 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 238 Base level 2, inputs: [5198(52MB) 5200(52MB) 5195(52MB) 5196(23MB)] 2024/11/19-15:26:54.838560 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005240.sst) /storage/mytopling/..//tmp/Topling-82038-jPk8pT 2024/11/19-15:26:55.198693 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-00 2024/11/19-15:26:55.198814 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00245/att-00/compact.done) = No such file or directory: 7.985 sec on http probe failed: 3, retry 2024/11/19-15:26:55.378787 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-01 2024/11/19-15:26:55.378903 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00242/att-01/compact.done) = No such file or directory: 13.039 sec on http probe failed: 5, giveup 2024/11/19-15:26:55.378944 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00242/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.782 sec, real 13.355 (submit 302.707 ms wait 13.039 s), input 253.782 MiB, send shutdown 2024/11/19-15:26:55.384968 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-01 2024/11/19-15:26:55.385237 139851158521600 [dcompact_etcd.cc:1773] job-00242/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:55.498822 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005240.sst): first pass time = 0.66's, 203.290'MB/sec 2024/11/19-15:26:55.498817 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005240.sst: sumWaitingMem = 0.012 GB, sumWorkingMem = 0.006 GB, nltTrie workingMem = 0.0120 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:26:55.498861 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005240.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.019 GB, dictZip workingMem = 0.0065 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:26:55.582619 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005240.sst): index pass time = 0.08's, 177.052'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 926853 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:26:55.681852 139851158521600 [dcompact_etcd.cc:1163] job-00242/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.297 sec, input = 253.782 MiB 2024/11/19-15:26:55.741938 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005239.sst): old prealloc_size = 147639500, real_size = 36234864 2024/11/19-15:26:55.742399 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005239.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924301 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1080 GB zip-val = 0.0342 GB avg-val = 116.80 avg-zval = 37.05 2024/11/19-15:26:55.742483 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005239.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 1.597 sec, Key+Value = 0.134 GB 2024/11/19-15:26:56.339608 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-01 2024/11/19-15:26:56.339743 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00244/att-01/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:26:56.371733 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005239.sst): second pass time = 1.60's, 67.613'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.63's, 58.896'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3944.301'MB/sec (index lex order gen) rebuild zvType time = 0.46's, 0.499'MB/sec write SST data time = 0.17's, 222.489'MB/sec dict compress time = 0.00's, 1386.342'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 12503.776'MB/sec zip my value time = 1.60's, unzip length = 0.108'GB zip my value throughput = 67.613'MB/sec zip pipeline throughput = 67.613'MB/sec entries = 924301 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 37.05 usrkeys = 924301 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 37.05 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 10 UnZipSize{ index = 0.0185 GB value = 0.1080 GB dict = 1.08 MB all = 0.1338 GB } __ZipSize{ index = 0.0018 GB value = 0.0342 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5126 value = 3.1526 dict = 1.00 all = 3.6085 } Zip/UnZip{ index = 0.0951 value = 0.3172 dict = 1.00 all = 0.2771 } ---------------------------- total value len = 16.506217 GB avg = 0.259 KB (by entry num) total key len = 1.646157 GB avg = 0.026 KB total ukey len = 1.135890 GB avg = 0.018 KB total ukey num = 0.063783396 Billion total entry num = 0.063783396 Billion write speed all = 36.348108374 MB/sec (with seq num) write speed all = 35.326355184 MB/sec (without seq num) 2024/11/19-15:26:56.373970 139851137550080 [compaction_job.cc:1948] [default] [JOB 238] Generated table #5239: 924301 keys, 37091201 bytes, temperature: kUnknown 2024/11/19-15:26:56.374039 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001216374005, "cf_name": "default", "job": 238, "event": "table_file_creation", "file_number": 5239, "file_size": 37091201, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34245152, "index_size": 1758464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25880428, "raw_average_key_size": 28, "raw_value_size": 107961522, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 924301, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001213, "oldest_key_time": 0, "file_creation_time": 1732001213, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5239, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:56.441070 139852362282752 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 12952258 2024/11/19-15:26:56.444947 139851137550080 [compaction_job.cc:703] job-00238: subcompact[0], size: 0.074331 G, files: /storage/mytopling/.rocksdb [005238,005239] 2024/11/19-15:26:56.448088 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 13352843 2024/11/19-15:26:56.452084 139851137550080 (Original Log Time 2024/11/19-15:26:51.337943) EVENT_LOG_v1 {"time_micros": 1732001211337904, "job": 238, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [5193, 5194, 5197, 5199], "score": 3.07839, "input_data_size": 221718240, "oldest_snapshot_seqno": 148685929} 2024/11/19-15:26:56.452099 139851137550080 (Original Log Time 2024/11/19-15:26:56.447576) [compaction_job.cc:2022] [default] [JOB 238] Compacted 4@2 files to L3 => 74330632 bytes 2024/11/19-15:26:56.452105 139851137550080 (Original Log Time 2024/11/19-15:26:56.448042) [compaction_job.cc:1224] [default] compacted to: files[7 45 84 64 0 0 0] max score 2.54 (112 files need compaction), MB/sec: 43.4 rd, 14.6 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(211.4, 0.0 +0.0 blob) out(70.9 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 1851252, records dropped: 0 output_compression: Snappy 2024/11/19-15:26:56.452134 139851137550080 (Original Log Time 2024/11/19-15:26:56.448080) EVENT_LOG_v1 {"time_micros": 1732001216448062, "cf": "default", "job": 238, "event": "compaction_finished", "compaction_time_micros": 5107024, "compaction_time_cpu_micros": 3992283, "output_level": 3, "num_output_files": 2, "total_output_size": 74330632, "num_input_records": 1851252, "num_output_records": 1851252, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [7, 45, 84, 64, 0, 0, 0]} 2024/11/19-15:26:56.548954 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005178.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:56.548999 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001216548994, "job": 238, "event": "table_file_deletion", "file_number": 5178} 2024/11/19-15:26:56.564581 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005179.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:56.564618 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001216564614, "job": 238, "event": "table_file_deletion", "file_number": 5179} 2024/11/19-15:26:56.564823 139851137550080 [compaction_job.cc:2437] [default] [JOB 247] Compacting 4@2 files to L3, score 2.50, subcompactions 1 : 1 2024/11/19-15:26:56.564839 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 245 Base level 2, inputs: [4797(62MB) 4800(63MB) 4801(61MB) 4802(63MB)] 2024/11/19-15:26:56.888716 139851137550080 [dcompact_etcd.cc:1163] job-00247/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.323 sec, input = 253.893 MiB 2024/11/19-15:26:57.060518 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005240.sst): old prealloc_size = 147639500, real_size = 36352960 2024/11/19-15:26:57.060972 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005240.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926853 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.81 avg-zval = 37.08 2024/11/19-15:26:57.061068 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005240.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 1.562 sec, Key+Value = 0.134 GB 2024/11/19-15:26:57.207067 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-00 2024/11/19-15:26:57.207143 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00245/att-00/compact.done) = No such file or directory: 10.001 sec on http probe failed: 4, retry 2024/11/19-15:26:57.209747 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-02 2024/11/19-15:26:57.209804 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00241/att-02/compact.done) = No such file or directory: 3.952 sec on http probe failed: 1, retry 2024/11/19-15:26:57.592366 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005240.sst): second pass time = 1.56's, 69.329'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.53's, 69.967'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3997.016'MB/sec (index lex order gen) rebuild zvType time = 0.44's, 0.532'MB/sec write SST data time = 0.10's, 388.506'MB/sec dict compress time = 0.00's, 1426.187'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 11174.072'MB/sec zip my value time = 1.56's, unzip length = 0.108'GB zip my value throughput = 69.329'MB/sec zip pipeline throughput = 69.329'MB/sec entries = 926853 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 37.08 usrkeys = 926853 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 37.08 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5813 value = 3.1501 dict = 1.00 all = 3.6069 } Zip/UnZip{ index = 0.0945 value = 0.3175 dict = 1.00 all = 0.2772 } ---------------------------- total value len = 16.614483 GB avg = 0.257 KB (by entry num) total key len = 1.672109 GB avg = 0.026 KB total ukey len = 1.154427 GB avg = 0.018 KB total ukey num = 0.064710249 Billion total entry num = 0.064710249 Billion write speed all = 36.527583718 MB/sec (with seq num) write speed all = 35.493510658 MB/sec (without seq num) 2024/11/19-15:26:57.593843 139850978166528 [compaction_job.cc:1948] [default] [JOB 240] Generated table #5240: 926853 keys, 37210917 bytes, temperature: kUnknown 2024/11/19-15:26:57.593896 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001217593871, "cf_name": "default", "job": 240, "event": "table_file_creation", "file_number": 5240, "file_size": 37210917, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34369200, "index_size": 1751872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25951884, "raw_average_key_size": 28, "raw_value_size": 108265974, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926853, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001214, "oldest_key_time": 0, "file_creation_time": 1732001214, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5240, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:57.614404 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005241.sst) /storage/mytopling/..//tmp/Topling-82038-mfmwZj 2024/11/19-15:26:57.635504 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-00 2024/11/19-15:26:57.635581 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00246/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:26:57.847947 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005241.sst): first pass time = 0.23's, 412.952'MB/sec 2024/11/19-15:26:57.847950 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005241.sst: sumWaitingMem = 0.009 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0087 GB, waited 0.000 sec, Key+Value = 0.096 GB 2024/11/19-15:26:57.847985 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005241.sst: sumWaitingMem = 0.005 GB, sumWorkingMem = 0.009 GB, dictZip workingMem = 0.0047 GB, waited 0.000 sec, Key+Value = 0.096 GB 2024/11/19-15:26:57.898069 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005241.sst): index pass time = 0.05's, 212.830'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 666128 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0107 GB zip-key = 0.0013 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:26:58.354648 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-01 2024/11/19-15:26:58.354744 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00244/att-01/compact.done) = No such file or directory: 5.188 sec on http probe failed: 2, retry 2024/11/19-15:26:58.845866 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005241.sst): old prealloc_size = 147639500, real_size = 26205920 2024/11/19-15:26:58.846219 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005241.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 666128 prefix = 4 raw-key = 0.0107 GB zip-key = 0.0013 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0778 GB zip-val = 0.0248 GB avg-val = 116.77 avg-zval = 37.20 2024/11/19-15:26:58.846337 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005241.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0017 GB, waited 0.998 sec, Key+Value = 0.096 GB 2024/11/19-15:26:59.215720 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-02 2024/11/19-15:26:59.215799 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00241/att-02/compact.done) = No such file or directory: 5.961 sec on http probe failed: 2, retry 2024/11/19-15:26:59.216153 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-00 2024/11/19-15:26:59.216193 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00245/att-00/compact.done) = No such file or directory: 12.009 sec on http probe failed: 5, giveup 2024/11/19-15:26:59.216206 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00245/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 158.601 sec, real 12.331 (submit 313.419 ms wait 12.009 s), input 158.601 MiB, send shutdown 2024/11/19-15:26:59.225270 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-00 2024/11/19-15:26:59.225488 139851127064320 [dcompact_etcd.cc:1773] job-00245/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:26:59.229455 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005241.sst): second pass time = 1.00's, 77.951'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.38's, 69.947'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3715.848'MB/sec (index lex order gen) rebuild zvType time = 0.31's, 0.533'MB/sec write SST data time = 0.07's, 378.864'MB/sec dict compress time = 0.00's, 1634.914'MB/sec z-dict build time = 0.00's, sample length = 0.784'MB, throughput = 8509.845'MB/sec zip my value time = 1.00's, unzip length = 0.078'GB zip my value throughput = 77.951'MB/sec zip pipeline throughput = 77.951'MB/sec entries = 666128 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 37.20 usrkeys = 666128 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 37.20 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0133 GB value = 0.0778 GB dict = 0.78 MB all = 0.0964 GB } __ZipSize{ index = 0.0013 GB value = 0.0248 GB dict = 0.78 MB all = 0.0268 GB } UnZip/Zip{ index = 10.5775 value = 3.1390 dict = 1.00 all = 3.5943 } Zip/UnZip{ index = 0.0945 value = 0.3186 dict = 1.00 all = 0.2782 } ---------------------------- total value len = 16.692267 GB avg = 0.255 KB (by entry num) total key len = 1.690761 GB avg = 0.026 KB total ukey len = 1.167750 GB avg = 0.018 KB total ukey num = 0.065376377 Billion total entry num = 0.065376377 Billion write speed all = 36.600525998 MB/sec (with seq num) write speed all = 35.559213382 MB/sec (without seq num) 2024/11/19-15:26:59.230243 139850978166528 [compaction_job.cc:1948] [default] [JOB 240] Generated table #5241: 666128 keys, 26830009 bytes, temperature: kUnknown 2024/11/19-15:26:59.230305 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001219230268, "cf_name": "default", "job": 240, "event": "table_file_creation", "file_number": 5241, "file_size": 26830009, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 24779664, "index_size": 1259520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 18651584, "raw_average_key_size": 28, "raw_value_size": 77783608, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 666128, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001217, "oldest_key_time": 0, "file_creation_time": 1732001217, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5241, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:26:59.245796 139850978166528 [compaction_job.cc:703] job-00240: subcompact[0], size: 0.064041 G, files: /storage/mytopling/.rocksdb [005240,005241] 2024/11/19-15:26:59.247949 139850978166528 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 13765817 2024/11/19-15:26:59.248076 139850978166528 (Original Log Time 2024/11/19-15:26:54.838238) EVENT_LOG_v1 {"time_micros": 1732001214838216, "job": 240, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [5198, 5200, 5195, 5196], "score": 2.91196, "input_data_size": 190769312, "oldest_snapshot_seqno": 148755002} 2024/11/19-15:26:59.248082 139850978166528 (Original Log Time 2024/11/19-15:26:59.247618) [compaction_job.cc:2022] [default] [JOB 240] Compacted 4@2 files to L3 => 64040926 bytes 2024/11/19-15:26:59.248085 139850978166528 (Original Log Time 2024/11/19-15:26:59.247907) [compaction_job.cc:1224] [default] compacted to: files[7 45 80 66 0 0 0] max score 2.54 (108 files need compaction), MB/sec: 43.3 rd, 14.5 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(181.9, 0.0 +0.0 blob) out(61.1 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 1592981, records dropped: 0 output_compression: Snappy 2024/11/19-15:26:59.248102 139850978166528 (Original Log Time 2024/11/19-15:26:59.247942) EVENT_LOG_v1 {"time_micros": 1732001219247917, "cf": "default", "job": 240, "event": "compaction_finished", "compaction_time_micros": 4407569, "compaction_time_cpu_micros": 3479500, "output_level": 3, "num_output_files": 2, "total_output_size": 64040926, "num_input_records": 1592981, "num_output_records": 1592981, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [7, 45, 80, 66, 0, 0, 0]} 2024/11/19-15:26:59.252792 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005081.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:59.252815 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001219252812, "job": 240, "event": "table_file_deletion", "file_number": 5081} 2024/11/19-15:26:59.257592 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005082.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:59.257613 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001219257611, "job": 240, "event": "table_file_deletion", "file_number": 5082} 2024/11/19-15:26:59.262231 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005083.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:59.262252 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001219262249, "job": 240, "event": "table_file_deletion", "file_number": 5083} 2024/11/19-15:26:59.267111 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005085.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:26:59.267131 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001219267129, "job": 240, "event": "table_file_deletion", "file_number": 5085} 2024/11/19-15:26:59.267711 139850978166528 [compaction_job.cc:2437] [default] [JOB 248] Compacting 4@2 files to L3, score 2.33, subcompactions 1 : 1 2024/11/19-15:26:59.267722 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 246 Base level 2, inputs: [4803(61MB) 4805(62MB) 4806(62MB) 4810(63MB)] 2024/11/19-15:26:59.500284 139851127064320 [dcompact_etcd.cc:1163] job-00245/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.275 sec, input = 158.601 MiB 2024/11/19-15:26:59.584851 139850978166528 [dcompact_etcd.cc:1163] job-00248/att-00: init time wrt = 0.001, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.316 sec, input = 253.822 MiB 2024/11/19-15:26:59.644222 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-00 2024/11/19-15:26:59.644302 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00246/att-00/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:27:00.362891 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-01 2024/11/19-15:27:00.362985 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00244/att-01/compact.done) = No such file or directory: 7.202 sec on http probe failed: 3, retry 2024/11/19-15:27:00.691069 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-02 2024/11/19-15:27:00.691150 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00242/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:27:01.224305 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-02 2024/11/19-15:27:01.224376 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00241/att-02/compact.done) = No such file or directory: 7.967 sec on http probe failed: 3, retry 2024/11/19-15:27:01.653645 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-00 2024/11/19-15:27:01.653721 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00246/att-00/compact.done) = No such file or directory: 9.018 sec on http probe failed: 3, retry 2024/11/19-15:27:01.894696 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-00 2024/11/19-15:27:01.894796 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00247/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:27:02.369220 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-01 2024/11/19-15:27:02.369303 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00244/att-01/compact.done) = No such file or directory: 9.210 sec on http probe failed: 4, retry 2024/11/19-15:27:02.699437 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-02 2024/11/19-15:27:02.699506 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00242/att-02/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:27:03.234820 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-02 2024/11/19-15:27:03.234916 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00241/att-02/compact.done) = No such file or directory: 9.975 sec on http probe failed: 4, retry 2024/11/19-15:27:03.475338 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-01 2024/11/19-15:27:03.475445 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00245/att-01/compact.done) = No such file or directory: 3.965 sec on http probe failed: 1, retry 2024/11/19-15:27:03.663243 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-00 2024/11/19-15:27:03.663330 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00246/att-00/compact.done) = No such file or directory: 11.027 sec on http probe failed: 4, retry 2024/11/19-15:27:03.903049 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-00 2024/11/19-15:27:03.903116 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00247/att-00/compact.done) = No such file or directory: 7.006 sec on http probe failed: 2, retry 2024/11/19-15:27:04.378276 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-01 2024/11/19-15:27:04.378396 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00244/att-01/compact.done) = No such file or directory: 11.217 sec on http probe failed: 5, giveup 2024/11/19-15:27:04.378415 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00244/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.952 sec, real 11.502 (submit 276.628 ms wait 11.217 s), input 126.952 MiB, send shutdown 2024/11/19-15:27:04.386872 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-01 2024/11/19-15:27:04.387092 139851179493120 [dcompact_etcd.cc:1773] job-00244/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:04.594494 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-00 2024/11/19-15:27:04.594563 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00248/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:27:04.659364 139851179493120 [dcompact_etcd.cc:1163] job-00244/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.272 sec, input = 126.952 MiB 2024/11/19-15:27:04.707631 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-02 2024/11/19-15:27:04.707684 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00242/att-02/compact.done) = No such file or directory: 9.018 sec on http probe failed: 3, retry 2024/11/19-15:27:05.243453 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-02 2024/11/19-15:27:05.243521 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00241/att-02/compact.done) = No such file or directory: 11.986 sec on http probe failed: 5, giveup 2024/11/19-15:27:05.243539 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00241/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 158.093 sec, real 12.292 (submit 297.283 ms wait 11.986 s), input 158.093 MiB, send shutdown 2024/11/19-15:27:05.248179 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00241/att-02 2024/11/19-15:27:05.248385 139851148035840 [dcompact_etcd.cc:1773] job-00241/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:05.248611 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00241) = success 2024/11/19-15:27:05.248617 139851148035840 [dcompact_etcd.cc:1773] job-00241/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:05.248634 139851148035840 (Original Log Time 2024/11/19-15:26:28.355149) EVENT_LOG_v1 {"time_micros": 1732001188355140, "job": 241, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4784, 4618, 4778, 4628], "score": 2.85136, "input_data_size": 163252572, "oldest_snapshot_seqno": -1} 2024/11/19-15:27:05.248642 139851148035840 [compaction_job.cc:2437] [default] [JOB 241] Compacting 4@2 files to L3, score 2.85, subcompactions 1 : 1 2024/11/19-15:27:05.248650 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 239 Base level 2, inputs: [4784(61MB) 4618(31MB) 4778(31MB) 4628(31MB)] 2024/11/19-15:27:05.251869 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005242.sst) /storage/mytopling/..//tmp/Topling-82038-4vCw4a 2024/11/19-15:27:05.453101 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005242.sst): first pass time = 0.20's, 667.037'MB/sec 2024/11/19-15:27:05.453107 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005242.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:27:05.453140 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005242.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0083 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:27:05.459745 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005242.sst): index pass time = 0.01's, 156.806'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 129434 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:27:05.484494 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-01 2024/11/19-15:27:05.484561 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00245/att-01/compact.done) = No such file or directory: 5.975 sec on http probe failed: 2, retry 2024/11/19-15:27:05.684938 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-00 2024/11/19-15:27:05.685099 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00246/att-00/compact.done) = No such file or directory: 13.039 sec on http probe failed: 5, giveup 2024/11/19-15:27:05.685133 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00246/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 222.689 sec, real 13.351 (submit 292.140 ms wait 13.039 s), input 222.689 MiB, send shutdown 2024/11/19-15:27:05.702804 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-00 2024/11/19-15:27:05.703095 139851169007360 [dcompact_etcd.cc:1773] job-00246/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:05.921013 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-00 2024/11/19-15:27:05.921128 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00247/att-00/compact.done) = No such file or directory: 9.016 sec on http probe failed: 3, retry 2024/11/19-15:27:06.020878 139851169007360 [dcompact_etcd.cc:1163] job-00246/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.318 sec, input = 222.689 MiB 2024/11/19-15:27:06.395396 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005242.sst): old prealloc_size = 147639500, real_size = 40347632 2024/11/19-15:27:06.395497 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005242.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129434 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1016.96 avg-zval = 309.78 2024/11/19-15:27:06.395549 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005242.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.942 sec, Key+Value = 0.134 GB 2024/11/19-15:27:06.510311 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005242.sst): second pass time = 0.94's, 139.699'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.11's, 362.898'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2693.186'MB/sec (index lex order gen) rebuild zvType time = 0.08's, 0.381'MB/sec write SST data time = 0.03's, 1392.430'MB/sec dict compress time = 0.00's, 1778.722'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 21695.324'MB/sec zip my value time = 0.94's, unzip length = 0.132'GB zip my value throughput = 139.699'MB/sec zip pipeline throughput = 139.699'MB/sec entries = 129434 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.96 avg-zval = 309.78 usrkeys = 129434 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1016.96 avg-zval = 309.78 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0982 value = 3.2828 dict = 1.00 all = 3.2190 } Zip/UnZip{ index = 0.1409 value = 0.3046 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 16.823896 GB avg = 0.257 KB (by entry num) total key len = 1.693349 GB avg = 0.026 KB total ukey len = 1.169303 GB avg = 0.018 KB total ukey num = 0.065505811 Billion total entry num = 0.065505811 Billion write speed all = 36.340949648 MB/sec (with seq num) write speed all = 35.312484191 MB/sec (without seq num) 2024/11/19-15:27:06.511735 139851148035840 [compaction_job.cc:1948] [default] [JOB 241] Generated table #5242: 129434 keys, 41695346 bytes, temperature: kUnknown 2024/11/19-15:27:06.511786 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001226511762, "cf_name": "default", "job": 241, "event": "table_file_creation", "file_number": 5242, "file_size": 41695346, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 40096288, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588680, "raw_average_key_size": 20, "raw_value_size": 131629175, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129434, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001225, "oldest_key_time": 0, "file_creation_time": 1732001225, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5242, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:06.532442 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005243.sst) /storage/mytopling/..//tmp/Topling-82038-eDF0JO 2024/11/19-15:27:06.574844 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005243.sst): first pass time = 0.04's, 744.424'MB/sec 2024/11/19-15:27:06.574852 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005243.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.032 GB 2024/11/19-15:27:06.574883 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005243.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, dictZip workingMem = 0.0018 GB, waited 0.000 sec, Key+Value = 0.032 GB 2024/11/19-15:27:06.574955 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005243.sst): index pass time = 0.00's,3252.980'MB/sec index type = UintIndex_AllOne usrkeys = 30430 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 6 raw-key = 0.0002 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.01 2024/11/19-15:27:06.600491 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-00 2024/11/19-15:27:06.600561 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00248/att-00/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:27:06.729788 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-02 2024/11/19-15:27:06.729889 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00242/att-02/compact.done) = No such file or directory: 11.028 sec on http probe failed: 4, retry 2024/11/19-15:27:06.845841 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005243.sst): old prealloc_size = 147639500, real_size = 9550408 2024/11/19-15:27:06.845882 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005243.sst): index type = UintIndex_AllOne , store type = DictZipBlobStore usrkeys = 30430 prefix = 4 raw-key = 0.0002 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.01 raw-val = 0.0309 GB zip-val = 0.0095 GB avg-val =1016.95 avg-zval = 313.59 2024/11/19-15:27:06.857175 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005243.sst): second pass time = 0.27's, 114.215'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.01's, 871.008'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 8.446'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 1.682'MB/sec write SST data time = 0.01's, 1456.644'MB/sec dict compress time = 0.00's, 1416.581'MB/sec z-dict build time = 0.00's, sample length = 0.304'MB, throughput = 5388.914'MB/sec zip my value time = 0.27's, unzip length = 0.031'GB zip my value throughput = 114.215'MB/sec zip pipeline throughput = 114.215'MB/sec entries = 30430 avg-key = 20.00 avg-zkey = 0.01 avg-val = 1016.95 avg-zval = 313.59 usrkeys = 30430 avg-key = 12.00 avg-zkey = 0.01 avg-val = 1016.95 avg-zval = 313.59 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 4 UnZipSize{ index = 0.0004 GB value = 0.0309 GB dict = 0.30 MB all = 0.0316 GB } __ZipSize{ index = 0.0000 GB value = 0.0095 GB dict = 0.30 MB all = 0.0098 GB } UnZip/Zip{ index =1901.8750 value = 3.2429 dict = 1.00 all = 3.2041 } Zip/UnZip{ index = 0.0005 value = 0.3084 dict = 1.00 all = 0.3121 } ---------------------------- total value len = 16.854842 GB avg = 0.257 KB (by entry num) total key len = 1.693958 GB avg = 0.026 KB total ukey len = 1.169668 GB avg = 0.018 KB total ukey num = 0.065536241 Billion total entry num = 0.065536241 Billion write speed all = 36.378113078 MB/sec (with seq num) write speed all = 35.349869803 MB/sec (without seq num) 2024/11/19-15:27:06.857864 139851148035840 [compaction_job.cc:1948] [default] [JOB 241] Generated table #5243: 30430 keys, 9848077 bytes, temperature: kUnknown 2024/11/19-15:27:06.857911 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001226857889, "cf_name": "default", "job": 241, "event": "table_file_creation", "file_number": 5243, "file_size": 9848077, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 9542480, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 608600, "raw_average_key_size": 20, "raw_value_size": 30945669, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 30430, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-DictZipBlobStore", "creation_time": 1732001226, "oldest_key_time": 0, "file_creation_time": 1732001226, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5243, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:06.863185 139851148035840 [compaction_job.cc:703] job-00241: subcompact[0], size: 0.051543 G, files: /storage/mytopling/.rocksdb [005242,005243] 2024/11/19-15:27:06.864455 139851148035840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 14191563 2024/11/19-15:27:06.864588 139851148035840 (Original Log Time 2024/11/19-15:27:05.248675) EVENT_LOG_v1 {"time_micros": 1732001225248661, "job": 241, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4784, 4618, 4778, 4628], "score": 2.85136, "input_data_size": 163252572, "oldest_snapshot_seqno": -1} 2024/11/19-15:27:06.864594 139851148035840 (Original Log Time 2024/11/19-15:27:06.864141) [compaction_job.cc:2022] [default] [JOB 241] Compacted 4@2 files to L3 => 51543423 bytes 2024/11/19-15:27:06.864597 139851148035840 (Original Log Time 2024/11/19-15:27:06.864426) [compaction_job.cc:1224] [default] compacted to: files[7 45 76 68 0 0 0] max score 2.54 (104 files need compaction), MB/sec: 101.1 rd, 31.9 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(155.7, 0.0 +0.0 blob) out(49.2 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 159864, records dropped: 0 output_compression: Snappy 2024/11/19-15:27:06.864612 139851148035840 (Original Log Time 2024/11/19-15:27:06.864448) EVENT_LOG_v1 {"time_micros": 1732001226864437, "cf": "default", "job": 241, "event": "compaction_finished", "compaction_time_micros": 1614522, "compaction_time_cpu_micros": 727529, "output_level": 3, "num_output_files": 2, "total_output_size": 51543423, "num_input_records": 159864, "num_output_records": 159864, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [7, 45, 76, 68, 0, 0, 0]} 2024/11/19-15:27:06.869274 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005086.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:06.869297 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001226869295, "job": 241, "event": "table_file_deletion", "file_number": 5086} 2024/11/19-15:27:06.873738 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005087.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:06.873758 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001226873756, "job": 241, "event": "table_file_deletion", "file_number": 5087} 2024/11/19-15:27:06.878286 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005089.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:06.878306 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001226878304, "job": 241, "event": "table_file_deletion", "file_number": 5089} 2024/11/19-15:27:06.882540 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005090.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:06.882559 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001226882557, "job": 241, "event": "table_file_deletion", "file_number": 5090} 2024/11/19-15:27:06.882692 139851148035840 [compaction_job.cc:2437] [default] [JOB 249] Compacting 4@2 files to L3, score 2.17, subcompactions 1 : 1 2024/11/19-15:27:06.882700 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 247 Base level 2, inputs: [4811(61MB) 4814(63MB) 4815(61MB) 4980(62MB)] 2024/11/19-15:27:07.196870 139851148035840 [dcompact_etcd.cc:1163] job-00249/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.314 sec, input = 253.643 MiB 2024/11/19-15:27:07.493432 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-01 2024/11/19-15:27:07.493548 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00245/att-01/compact.done) = No such file or directory: 7.984 sec on http probe failed: 3, retry 2024/11/19-15:27:07.838381 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-02 2024/11/19-15:27:07.838478 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00244/att-02/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:27:07.930414 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-00 2024/11/19-15:27:07.930479 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00247/att-00/compact.done) = No such file or directory: 11.033 sec on http probe failed: 4, retry 2024/11/19-15:27:08.609084 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-00 2024/11/19-15:27:08.609181 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00248/att-00/compact.done) = No such file or directory: 9.016 sec on http probe failed: 3, retry 2024/11/19-15:27:08.734592 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-02 2024/11/19-15:27:08.734657 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00242/att-02/compact.done) = No such file or directory: 13.048 sec on http probe failed: 5, giveup 2024/11/19-15:27:08.734673 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00242/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.782 sec, real 13.349 (submit 296.503 ms wait 13.048 s), input 253.782 MiB, send shutdown 2024/11/19-15:27:08.740837 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00242/att-02 2024/11/19-15:27:08.741064 139851158521600 [dcompact_etcd.cc:1773] job-00242/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:08.741309 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00242) = success 2024/11/19-15:27:08.741316 139851158521600 [dcompact_etcd.cc:1773] job-00242/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:08.741333 139851158521600 (Original Log Time 2024/11/19-15:26:28.678701) EVENT_LOG_v1 {"time_micros": 1732001188678693, "job": 242, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4788, 4789, 4790, 4791], "score": 2.83112, "input_data_size": 262147350, "oldest_snapshot_seqno": -1} 2024/11/19-15:27:08.741340 139851158521600 [compaction_job.cc:2437] [default] [JOB 242] Compacting 4@2 files to L3, score 2.83, subcompactions 1 : 1 2024/11/19-15:27:08.741347 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 240 Base level 2, inputs: [4788(63MB) 4789(61MB) 4790(63MB) 4791(61MB)] 2024/11/19-15:27:08.743534 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005244.sst) /storage/mytopling/..//tmp/Topling-82038-XPx4nD 2024/11/19-15:27:08.961689 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005244.sst): first pass time = 0.22's, 615.297'MB/sec 2024/11/19-15:27:08.961691 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005244.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:27:08.961730 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005244.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0083 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:27:08.968154 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005244.sst): index pass time = 0.01's, 161.247'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 129427 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:27:09.521984 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-01 2024/11/19-15:27:09.522113 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00245/att-01/compact.done) = No such file or directory: 9.999 sec on http probe failed: 4, retry 2024/11/19-15:27:09.849085 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-02 2024/11/19-15:27:09.849188 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00244/att-02/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:27:09.895002 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005244.sst): old prealloc_size = 147639500, real_size = 40354672 2024/11/19-15:27:09.895112 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005244.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129427 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.02 avg-zval = 309.85 2024/11/19-15:27:09.895155 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005244.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.933 sec, Key+Value = 0.134 GB 2024/11/19-15:27:09.939820 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-00 2024/11/19-15:27:09.939879 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00247/att-00/compact.done) = No such file or directory: 13.042 sec on http probe failed: 5, giveup 2024/11/19-15:27:09.939896 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00247/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.893 sec, real 13.374 (submit 323.272 ms wait 13.042 s), input 253.893 MiB, send shutdown 2024/11/19-15:27:09.944363 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-00 2024/11/19-15:27:09.944557 139851137550080 [dcompact_etcd.cc:1773] job-00247/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:10.012459 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005244.sst): second pass time = 0.93's, 141.045'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 355.096'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2340.228'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 0.375'MB/sec write SST data time = 0.03's, 1341.193'MB/sec dict compress time = 0.00's, 1817.460'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 27714.223'MB/sec zip my value time = 0.93's, unzip length = 0.132'GB zip my value throughput = 141.045'MB/sec zip pipeline throughput = 141.045'MB/sec entries = 129427 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.02 avg-zval = 309.85 usrkeys = 129427 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.02 avg-zval = 309.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0979 value = 3.2823 dict = 1.00 all = 3.2185 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 16.986471 GB avg = 0.259 KB (by entry num) total key len = 1.696546 GB avg = 0.026 KB total ukey len = 1.171221 GB avg = 0.018 KB total ukey num = 0.065665668 Billion total entry num = 0.065665668 Billion write speed all = 36.415993313 MB/sec (with seq num) write speed all = 35.392055701 MB/sec (without seq num) 2024/11/19-15:27:10.013781 139851158521600 [compaction_job.cc:1948] [default] [JOB 242] Generated table #5244: 129427 keys, 41702079 bytes, temperature: kUnknown 2024/11/19-15:27:10.013832 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001230013808, "cf_name": "default", "job": 242, "event": "table_file_creation", "file_number": 5244, "file_size": 41702079, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 40103328, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588540, "raw_average_key_size": 20, "raw_value_size": 131629385, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129427, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001228, "oldest_key_time": 0, "file_creation_time": 1732001228, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5244, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:10.035235 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005245.sst) /storage/mytopling/..//tmp/Topling-82038-61ulpb 2024/11/19-15:27:10.223177 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005245.sst): first pass time = 0.19's, 701.829'MB/sec 2024/11/19-15:27:10.223183 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005245.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.132 GB 2024/11/19-15:27:10.223218 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005245.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0081 GB, waited 0.000 sec, Key+Value = 0.132 GB 2024/11/19-15:27:10.232692 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005245.sst): index pass time = 0.01's, 107.429'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 127183 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:27:10.251532 139851137550080 [dcompact_etcd.cc:1163] job-00247/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.307 sec, input = 253.893 MiB 2024/11/19-15:27:10.627279 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-00 2024/11/19-15:27:10.627373 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00248/att-00/compact.done) = No such file or directory: 11.027 sec on http probe failed: 4, retry 2024/11/19-15:27:11.029917 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-01 2024/11/19-15:27:11.030017 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00246/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:27:11.089395 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005245.sst): old prealloc_size = 147639500, real_size = 39657584 2024/11/19-15:27:11.089495 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005245.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 127183 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1293 GB zip-val = 0.0394 GB avg-val =1017.03 avg-zval = 309.87 2024/11/19-15:27:11.089537 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005245.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.866 sec, Key+Value = 0.132 GB 2024/11/19-15:27:11.205217 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005245.sst): second pass time = 0.87's, 149.336'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 353.906'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2702.424'MB/sec (index lex order gen) rebuild zvType time = 0.08's, 0.376'MB/sec write SST data time = 0.03's, 1319.179'MB/sec dict compress time = 0.00's, 1515.111'MB/sec z-dict build time = 0.00's, sample length = 1.356'MB, throughput = 24597.790'MB/sec zip my value time = 0.87's, unzip length = 0.129'GB zip my value throughput = 149.336'MB/sec zip pipeline throughput = 149.336'MB/sec entries = 127183 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.03 avg-zval = 309.87 usrkeys = 127183 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.03 avg-zval = 309.87 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0015 GB value = 0.1293 GB dict = 1.36 MB all = 0.1319 GB } __ZipSize{ index = 0.0002 GB value = 0.0394 GB dict = 1.36 MB all = 0.0410 GB } UnZip/Zip{ index = 7.0973 value = 3.2821 dict = 1.00 all = 3.2182 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 17.115819 GB avg = 0.260 KB (by entry num) total key len = 1.699090 GB avg = 0.026 KB total ukey len = 1.172747 GB avg = 0.018 KB total ukey num = 0.065792851 Billion total entry num = 0.065792851 Billion write speed all = 36.588008773 MB/sec (with seq num) write speed all = 35.564467558 MB/sec (without seq num) 2024/11/19-15:27:11.206764 139851158521600 [compaction_job.cc:1948] [default] [JOB 242] Generated table #5245: 127183 keys, 40983163 bytes, temperature: kUnknown 2024/11/19-15:27:11.206816 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001231206792, "cf_name": "default", "job": 242, "event": "table_file_creation", "file_number": 5245, "file_size": 40983163, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 39410592, "index_size": 215040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2543660, "raw_average_key_size": 20, "raw_value_size": 129348409, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 127183, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001230, "oldest_key_time": 0, "file_creation_time": 1732001230, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5245, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:11.229182 139851158521600 [compaction_job.cc:703] job-00242: subcompact[0], size: 0.082685 G, files: /storage/mytopling/.rocksdb [005244,005245] 2024/11/19-15:27:11.230894 139851158521600 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 14630477 2024/11/19-15:27:11.231039 139851158521600 (Original Log Time 2024/11/19-15:27:08.741377) EVENT_LOG_v1 {"time_micros": 1732001228741359, "job": 242, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4788, 4789, 4790, 4791], "score": 2.83112, "input_data_size": 262147350, "oldest_snapshot_seqno": -1} 2024/11/19-15:27:11.231054 139851158521600 (Original Log Time 2024/11/19-15:27:11.230569) [compaction_job.cc:2022] [default] [JOB 242] Compacted 4@2 files to L3 => 82685242 bytes 2024/11/19-15:27:11.231058 139851158521600 (Original Log Time 2024/11/19-15:27:11.230860) [compaction_job.cc:1224] [default] compacted to: files[7 45 72 70 0 0 0] max score 2.54 (100 files need compaction), MB/sec: 105.4 rd, 33.2 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(250.0, 0.0 +0.0 blob) out(78.9 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 256610, records dropped: 0 output_compression: Snappy 2024/11/19-15:27:11.231076 139851158521600 (Original Log Time 2024/11/19-15:27:11.230886) EVENT_LOG_v1 {"time_micros": 1732001231230874, "cf": "default", "job": 242, "event": "compaction_finished", "compaction_time_micros": 2487819, "compaction_time_cpu_micros": 1082659, "output_level": 3, "num_output_files": 2, "total_output_size": 82685242, "num_input_records": 256610, "num_output_records": 256610, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [7, 45, 72, 70, 0, 0, 0]} 2024/11/19-15:27:11.238948 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005174.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:11.238975 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001231238972, "job": 242, "event": "table_file_deletion", "file_number": 5174} 2024/11/19-15:27:11.247121 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005175.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:11.247146 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001231247143, "job": 242, "event": "table_file_deletion", "file_number": 5175} 2024/11/19-15:27:11.256110 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005183.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:11.256133 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001231256131, "job": 242, "event": "table_file_deletion", "file_number": 5183} 2024/11/19-15:27:11.265081 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005184.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:11.265104 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001231265101, "job": 242, "event": "table_file_deletion", "file_number": 5184} 2024/11/19-15:27:11.265243 139851158521600 [compaction_job.cc:2437] [default] [JOB 250] Compacting 4@2 files to L3, score 2.00, subcompactions 1 : 1 2024/11/19-15:27:11.265252 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 248 Base level 2, inputs: [4981(32MB) 4982(62MB) 4983(62MB) 4986(62MB)] 2024/11/19-15:27:11.532802 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-01 2024/11/19-15:27:11.532903 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00245/att-01/compact.done) = No such file or directory: 12.022 sec on http probe failed: 5, giveup 2024/11/19-15:27:11.532929 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00245/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 158.601 sec, real 12.307 (submit 274.700 ms wait 12.022 s), input 158.601 MiB, send shutdown 2024/11/19-15:27:11.541357 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-01 2024/11/19-15:27:11.541574 139851127064320 [dcompact_etcd.cc:1773] job-00245/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:11.546116 139851158521600 [dcompact_etcd.cc:1163] job-00250/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 224.039 MiB 2024/11/19-15:27:11.805753 139851127064320 [dcompact_etcd.cc:1163] job-00245/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.264 sec, input = 158.601 MiB 2024/11/19-15:27:11.860647 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-02 2024/11/19-15:27:11.860709 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00244/att-02/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:27:12.020937 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #5246. Immutable memtables: 0. 2024/11/19-15:27:12.021063 139852362282752 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 7 level-0 files rate 14191562 2024/11/19-15:27:12.021146 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 251] Syncing log #5231 2024/11/19-15:27:12.021227 139850957195008 (Original Log Time 2024/11/19-15:27:12.021132) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:27:12.021235 139850957195008 [flush_job.cc:879] [default] [JOB 251] Flushing memtable with next log file: 5246 2024/11/19-15:27:12.021255 139850957195008 EVENT_LOG_v1 {"time_micros": 1732001232021250, "job": 251, "event": "flush_started", "num_memtables": 1, "num_entries": 1232632, "num_deletes": 0, "total_data_size": 481296893, "memory_usage": 540016008, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:27:12.021259 139850957195008 [flush_job.cc:910] [default] [JOB 251] Level-0 flush table #5247: started 2024/11/19-15:27:12.021625 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000032.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/005247.sst in ConvertToSST, mem_size = 514.999 M, time = 357.124 us 2024/11/19-15:27:12.202559 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-00 2024/11/19-15:27:12.202725 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00249/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:27:12.500511 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/005247.sst): fsize = 515.001 M, time(ms): open: 0.035, seek: 0.014, finish: 0.036, meta: 0.013, Flush: 0.011, sync: 478.751, close: 0.009, all: 478.869 2024/11/19-15:27:12.501672 139850957195008 [flush_job.cc:1083] [default] [JOB 251] Flush lasted 480444 microseconds, and 289454 cpu microseconds. 2024/11/19-15:27:12.523585 139850957195008 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 8 level-0 files rate 8514937 2024/11/19-15:27:12.523659 139850957195008 (Original Log Time 2024/11/19-15:27:12.500561) [flush_job.cc:1029] [default] [JOB 251] Level-0 flush table #5247: 540017434 bytes OK (needs compaction) 2024/11/19-15:27:12.523664 139850957195008 (Original Log Time 2024/11/19-15:27:12.501690) [memtable_list.cc:560] [default] Level-0 commit table #5247 started 2024/11/19-15:27:12.523667 139850957195008 (Original Log Time 2024/11/19-15:27:12.523555) [memtable_list.cc:759] [default] Level-0 commit table #5247: memtable #1 done 2024/11/19-15:27:12.523670 139850957195008 (Original Log Time 2024/11/19-15:27:12.523579) EVENT_LOG_v1 {"time_micros": 1732001232523573, "job": 251, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [8, 45, 72, 70, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:27:12.523673 139850957195008 (Original Log Time 2024/11/19-15:27:12.523633) [db_impl_compaction_flush.cc:310] [default] Level summary: files[8 45 72 70 0 0 0] max score 2.90 (97 files need compaction) 2024/11/19-15:27:12.635110 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-00 2024/11/19-15:27:12.635197 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00248/att-00/compact.done) = No such file or directory: 13.043 sec on http probe failed: 5, giveup 2024/11/19-15:27:12.635215 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00248/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.822 sec, real 13.367 (submit 316.491 ms wait 13.043 s), input 253.822 MiB, send shutdown 2024/11/19-15:27:12.643758 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-00 2024/11/19-15:27:12.643977 139850978166528 [dcompact_etcd.cc:1773] job-00248/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:12.947960 139850978166528 [dcompact_etcd.cc:1163] job-00248/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.304 sec, input = 253.822 MiB 2024/11/19-15:27:13.038160 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-01 2024/11/19-15:27:13.038221 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00246/att-01/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:27:13.869341 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-02 2024/11/19-15:27:13.869417 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00244/att-02/compact.done) = No such file or directory: 9.201 sec on http probe failed: 4, retry 2024/11/19-15:27:14.211500 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-00 2024/11/19-15:27:14.211599 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00249/att-00/compact.done) = No such file or directory: 7.006 sec on http probe failed: 2, retry 2024/11/19-15:27:15.043230 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-01 2024/11/19-15:27:15.043303 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00246/att-01/compact.done) = No such file or directory: 9.017 sec on http probe failed: 3, retry 2024/11/19-15:27:15.261283 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-01 2024/11/19-15:27:15.261369 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00247/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:27:15.776534 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-02 2024/11/19-15:27:15.776624 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00245/att-02/compact.done) = No such file or directory: 3.965 sec on http probe failed: 1, retry 2024/11/19-15:27:15.875593 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-02 2024/11/19-15:27:15.875658 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00244/att-02/compact.done) = No such file or directory: 11.210 sec on http probe failed: 5, giveup 2024/11/19-15:27:15.875674 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00244/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.952 sec, real 11.489 (submit 272.205 ms wait 11.210 s), input 126.952 MiB, send shutdown 2024/11/19-15:27:15.884198 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00244/att-02 2024/11/19-15:27:15.884440 139851179493120 [dcompact_etcd.cc:1773] job-00244/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:15.884683 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00244) = success 2024/11/19-15:27:15.884688 139851179493120 [dcompact_etcd.cc:1773] job-00244/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:15.884706 139851179493120 (Original Log Time 2024/11/19-15:26:41.315416) EVENT_LOG_v1 {"time_micros": 1732001201315409, "job": 244, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5097, 5099, 5101, 5103], "score": 2.77781, "input_data_size": 134099136, "oldest_snapshot_seqno": 149750226} 2024/11/19-15:27:15.884714 139851179493120 [compaction_job.cc:2437] [default] [JOB 244] Compacting 4@1 files to L2, score 2.78, subcompactions 1 : 1 2024/11/19-15:27:15.884721 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 242 Base level 1, inputs: [5097(31MB) 5099(31MB) 5101(31MB) 5103(31MB)] 2024/11/19-15:27:15.884961 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005248.sst) /storage/mytopling/..//tmp/Topling-82038-yuLa0a 2024/11/19-15:27:15.940075 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005248.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:27:15.940189 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005248.sst): index pass time = 0.00's,6341.385'MB/sec index type = UintIndex_AllOne usrkeys = 64717 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:27:16.008948 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005248.sst): first pass time = 0.12's, 541.322'MB/sec 2024/11/19-15:27:16.009005 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005248.sst): old prealloc_size = 73819750, real_size = 66042232 2024/11/19-15:27:16.009029 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005248.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64717 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.97 avg-zval =1020.22 2024/11/19-15:27:16.064748 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005248.sst): second pass time = 0.00's,220856312.081'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1184.804'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.177'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.409'MB/sec write SST data time = 0.02's, 4094.139'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 220856312.081'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64717 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.97 avg-zval = 1020.22 usrkeys = 64717 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.97 avg-zval = 1020.22 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.8125 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 17.181635 GB avg = 0.261 KB (by entry num) total key len = 1.700384 GB avg = 0.026 KB total ukey len = 1.173524 GB avg = 0.018 KB total ukey num = 0.065857568 Billion total entry num = 0.065857568 Billion write speed all = 36.374770758 MB/sec (with seq num) write speed all = 35.359814066 MB/sec (without seq num) 2024/11/19-15:27:16.068776 139851179493120 [compaction_job.cc:1948] [default] [JOB 244] Generated table #5248: 64717 keys, 66027018 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.068828 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236068805, "cf_name": "default", "job": 244, "event": "table_file_creation", "file_number": 5248, "file_size": 66027018, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025728, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294340, "raw_average_key_size": 20, "raw_value_size": 65815181, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64717, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001235, "oldest_key_time": 0, "file_creation_time": 1732001235, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5248, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:16.084446 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005249.sst) /storage/mytopling/..//tmp/Topling-82038-x62YuY 2024/11/19-15:27:16.136875 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005249.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:27:16.142037 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005249.sst): index pass time = 0.01's, 99.348'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63653 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:27:16.205881 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005249.sst): first pass time = 0.12's, 543.657'MB/sec 2024/11/19-15:27:16.205943 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005249.sst): old prealloc_size = 73819750, real_size = 65067904 2024/11/19-15:27:16.206010 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005249.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63653 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1017.02 avg-zval =1020.27 2024/11/19-15:27:16.206063 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005249.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.069 sec, Key+Value = 0.066 GB 2024/11/19-15:27:16.216419 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-00 2024/11/19-15:27:16.216510 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00249/att-00/compact.done) = No such file or directory: 9.015 sec on http probe failed: 3, retry 2024/11/19-15:27:16.267116 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005249.sst): second pass time = 0.00's,218703756.757'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1063.498'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1897.940'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.365'MB/sec write SST data time = 0.02's, 3703.606'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 218703756.757'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63653 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.02 avg-zval = 1020.27 usrkeys = 63653 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.02 avg-zval = 1020.27 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7932 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0651 GB } UnZip/Zip{ index = 7.0454 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 17.246371 GB avg = 0.262 KB (by entry num) total key len = 1.701657 GB avg = 0.026 KB total ukey len = 1.174288 GB avg = 0.018 KB total ukey num = 0.065921221 Billion total entry num = 0.065921221 Billion write speed all = 36.487707972 MB/sec (with seq num) write speed all = 35.472166210 MB/sec (without seq num) 2024/11/19-15:27:16.271162 139851179493120 [compaction_job.cc:1948] [default] [JOB 244] Generated table #5249: 63653 keys, 65053380 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.271210 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236271187, "cf_name": "default", "job": 244, "event": "table_file_creation", "file_number": 5249, "file_size": 65053380, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64943408, "index_size": 108416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1273060, "raw_average_key_size": 20, "raw_value_size": 64736312, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63653, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5249, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:16.287965 139851179493120 [compaction_job.cc:703] job-00244: subcompact[0], size: 0.131080 G, files: /storage/mytopling/.rocksdb [005248,005249] 2024/11/19-15:27:16.288686 139851179493120 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 8 level-0 files rate 5108962 2024/11/19-15:27:16.288839 139851179493120 (Original Log Time 2024/11/19-15:27:15.884748) EVENT_LOG_v1 {"time_micros": 1732001235884734, "job": 244, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5097, 5099, 5101, 5103], "score": 2.77781, "input_data_size": 134099136, "oldest_snapshot_seqno": 149750226} 2024/11/19-15:27:16.288846 139851179493120 (Original Log Time 2024/11/19-15:27:16.288367) [compaction_job.cc:2022] [default] [JOB 244] Compacted 4@1 files to L2 => 131080398 bytes 2024/11/19-15:27:16.288849 139851179493120 (Original Log Time 2024/11/19-15:27:16.288651) [compaction_job.cc:1224] [default] compacted to: files[8 41 74 70 0 0 0] max score 3.19 (99 files need compaction), MB/sec: 332.6 rd, 325.1 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128370, records dropped: 0 output_compression: Snappy 2024/11/19-15:27:16.288867 139851179493120 (Original Log Time 2024/11/19-15:27:16.288679) EVENT_LOG_v1 {"time_micros": 1732001236288666, "cf": "default", "job": 244, "event": "compaction_finished", "compaction_time_micros": 403229, "compaction_time_cpu_micros": 401409, "output_level": 2, "num_output_files": 2, "total_output_size": 131080398, "num_input_records": 128370, "num_output_records": 128370, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [8, 41, 74, 70, 0, 0, 0]} 2024/11/19-15:27:16.293280 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005092.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:16.293303 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236293301, "job": 244, "event": "table_file_deletion", "file_number": 5092} 2024/11/19-15:27:16.297380 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005093.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:16.297399 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236297397, "job": 244, "event": "table_file_deletion", "file_number": 5093} 2024/11/19-15:27:16.301504 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005094.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:16.301523 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236301521, "job": 244, "event": "table_file_deletion", "file_number": 5094} 2024/11/19-15:27:16.305508 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005096.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:16.305528 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236305526, "job": 244, "event": "table_file_deletion", "file_number": 5096} 2024/11/19-15:27:16.321427 139851179493120 [compaction_job.cc:2437] [default] [JOB 252] Compacting 8@0 + 41@1 files to L1, score 3.19, subcompactions 3 : 3 2024/11/19-15:27:16.321453 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 250 Base level 0, inputs: [5247(515MB) 5232(519MB) 5219(512MB) 5192(512MB) 5188(520MB) 5173(512MB) 5168(520MB) 5059(512MB)], [5105(31MB) 5107(31MB) 5109(31MB) 5110(31MB) 5111(31MB) 5112(31MB) 5114(31MB) 5115(31MB) 5116(31MB) 5117(31MB) 5119(31MB) 5121(31MB) 5122(31MB) 5124(31MB) 5125(31MB) 5126(31MB) 5127(31MB) 5129(31MB) 5130(31MB) 5131(31MB) 5132(31MB) 5134(31MB) 5135(31MB) 5136(31MB) 5138(31MB) 5139(31MB) 5140(31MB) 5141(31MB) 5143(31MB) 5144(31MB) 5145(31MB) 5147(31MB) 5149(31MB) 5151(31MB) 5152(31MB) 5153(31MB) 5154(31MB) 5155(31MB) 5156(31MB) 5157(31MB) 5158(15MB)] 2024/11/19-15:27:16.322026 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5250: 23 keys, 7923 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.322060 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236322041, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5250, "file_size": 7923, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 140552265, "largest_seqno": 150747464, "table_properties": {"data_size": 2752, "index_size": 4352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 368, "raw_average_key_size": 16, "raw_value_size": 2717, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 23, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5250, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.322710 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5253: 223 keys, 37880 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.322738 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236322723, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5253, "file_size": 37880, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139864614, "largest_seqno": 150874067, "table_properties": {"data_size": 26944, "index_size": 10112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4460, "raw_average_key_size": 20, "raw_value_size": 26930, "raw_average_value_size": 120, "num_data_blocks": 1, "num_entries": 223, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5253, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.353748 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5252: 32060 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.353810 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001236353789, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5252, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 133733511, "largest_seqno": 133795579, "table_properties": {"data_size": 32604672, "index_size": 919040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32604648, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5252, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.362572 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5254: 49875 keys, 33507200 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.362615 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236362595, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5254, "file_size": 33507200, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139814828, "largest_seqno": 146922506, "table_properties": {"data_size": 32158336, "index_size": 1348032, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1197000, "raw_average_key_size": 24, "raw_value_size": 32158282, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49875, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5254, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.385665 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5255: 32053 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.385709 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001236385690, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5255, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 133795580, "largest_seqno": 133890578, "table_properties": {"data_size": 32598528, "index_size": 925888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32598499, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5255, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.401079 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5256: 49847 keys, 33507072 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.401123 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236401103, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5256, "file_size": 33507072, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 140431222, "largest_seqno": 141504083, "table_properties": {"data_size": 32157760, "index_size": 1348480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196328, "raw_average_key_size": 24, "raw_value_size": 32157721, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49847, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5256, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.417305 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5257: 32055 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.417349 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001236417329, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5257, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 133890579, "largest_seqno": 134600926, "table_properties": {"data_size": 32600064, "index_size": 923776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32600053, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5257, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.436791 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5251: 321995 keys, 25302467 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.436859 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001236436837, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5251, "file_size": 25302467, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 145384048, "largest_seqno": 150820309, "table_properties": {"data_size": 17386176, "index_size": 7915456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6439900, "raw_average_key_size": 20, "raw_value_size": 17386127, "raw_average_value_size": 53, "num_data_blocks": 1, "num_entries": 321995, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5251, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.441289 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5258: 49833 keys, 33506944 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.441341 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236441315, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5258, "file_size": 33506944, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141192130, "largest_seqno": 142469131, "table_properties": {"data_size": 32159488, "index_size": 1346624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1195992, "raw_average_key_size": 24, "raw_value_size": 32159429, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49833, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5258, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.448495 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5259: 32063 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.448536 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001236448517, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5259, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134600927, "largest_seqno": 134678850, "table_properties": {"data_size": 32608768, "index_size": 914880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641260, "raw_average_key_size": 20, "raw_value_size": 32608712, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32063, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5259, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.479889 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5262: 32057 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.479955 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001236479916, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5262, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134678851, "largest_seqno": 134761904, "table_properties": {"data_size": 32599232, "index_size": 925248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32599220, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5262, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.485210 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5261: 49874 keys, 33506688 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.485251 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236485232, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5261, "file_size": 33506688, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141981030, "largest_seqno": 143229262, "table_properties": {"data_size": 32157760, "index_size": 1348096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196976, "raw_average_key_size": 24, "raw_value_size": 32157749, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49874, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5261, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.511655 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5263: 32152 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.511700 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001236511680, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5263, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134723929, "largest_seqno": 134813940, "table_properties": {"data_size": 32697600, "index_size": 826752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643040, "raw_average_key_size": 20, "raw_value_size": 32697597, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32152, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5263, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.514413 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5260: 199800 keys, 5186830 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.514455 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001236514435, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5260, "file_size": 5186830, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139863713, "largest_seqno": 150874065, "table_properties": {"data_size": 199808, "index_size": 4986176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4795200, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 199800, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5260, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.551648 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-00 2024/11/19-15:27:16.551746 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00250/att-00/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:27:16.626702 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5264: 49834 keys, 33505664 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.626748 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236626727, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5264, "file_size": 33505664, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 142948754, "largest_seqno": 143985949, "table_properties": {"data_size": 32154688, "index_size": 1350144, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196016, "raw_average_key_size": 24, "raw_value_size": 32154660, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49834, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5264, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.657389 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5265: 32151 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.657437 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001236657416, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5265, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134724052, "largest_seqno": 134786211, "table_properties": {"data_size": 32696832, "index_size": 827520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32696774, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5265, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.665060 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5267: 49916 keys, 33506688 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.665118 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236665088, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5267, "file_size": 33506688, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 143945831, "largest_seqno": 144657697, "table_properties": {"data_size": 32156736, "index_size": 1349120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1197984, "raw_average_key_size": 24, "raw_value_size": 32156690, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49916, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5267, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.689001 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5268: 32071 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.689046 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001236689026, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5268, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134786212, "largest_seqno": 134847295, "table_properties": {"data_size": 32615936, "index_size": 907968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641420, "raw_average_key_size": 20, "raw_value_size": 32615918, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32071, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5268, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.726387 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5269: 49901 keys, 33507008 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.726434 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236726414, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5269, "file_size": 33507008, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 144657698, "largest_seqno": 145462141, "table_properties": {"data_size": 32157824, "index_size": 1348352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1197624, "raw_average_key_size": 24, "raw_value_size": 32157782, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49901, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5269, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.736682 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5270: 32058 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.736726 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001236736705, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5270, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134847296, "largest_seqno": 134950344, "table_properties": {"data_size": 32602240, "index_size": 921344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602235, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5270, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.774499 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5271: 49869 keys, 33507072 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.774552 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236774531, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5271, "file_size": 33507072, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 145453316, "largest_seqno": 146455991, "table_properties": {"data_size": 32158208, "index_size": 1348032, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196856, "raw_average_key_size": 24, "raw_value_size": 32158175, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49869, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5271, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.774641 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5266: 666000 keys, 29463747 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.774686 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001236774666, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5266, "file_size": 29463747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139820830, "largest_seqno": 150833311, "table_properties": {"data_size": 13186816, "index_size": 16276096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 15984000, "raw_average_key_size": 24, "raw_value_size": 13186800, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 666000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5266, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.785905 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5272: 32059 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.785955 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001236785932, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5272, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 134950345, "largest_seqno": 135666483, "table_properties": {"data_size": 32603712, "index_size": 920000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603696, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5272, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.824627 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5273: 49836 keys, 33506752 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.824680 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236824658, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5273, "file_size": 33506752, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 146133302, "largest_seqno": 147450614, "table_properties": {"data_size": 32159232, "index_size": 1346688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196064, "raw_average_key_size": 24, "raw_value_size": 32159206, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49836, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5273, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.824725 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5275: 32097 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.824769 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001236824749, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5275, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 135666484, "largest_seqno": 135752420, "table_properties": {"data_size": 32641920, "index_size": 881984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641940, "raw_average_key_size": 20, "raw_value_size": 32641885, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32097, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5275, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.866537 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5277: 32057 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.866591 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001236866569, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5277, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 135752421, "largest_seqno": 135814486, "table_properties": {"data_size": 32602560, "index_size": 921216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602509, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5277, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.877752 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5276: 49827 keys, 33506816 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.877803 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236877782, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5276, "file_size": 33506816, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 146938515, "largest_seqno": 148208095, "table_properties": {"data_size": 32157696, "index_size": 1348288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1195848, "raw_average_key_size": 24, "raw_value_size": 32157693, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49827, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5276, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.897736 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5274: 231253 keys, 33325251 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.897788 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001236897767, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5274, "file_size": 33325251, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139833832, "largest_seqno": 140383247, "table_properties": {"data_size": 27031168, "index_size": 6293248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6475084, "raw_average_key_size": 28, "raw_value_size": 27031138, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231253, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5274, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.907150 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5278: 32057 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.907197 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001236907176, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5278, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 135721858, "largest_seqno": 135821870, "table_properties": {"data_size": 32602944, "index_size": 920832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602905, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5278, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.927980 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5279: 49841 keys, 33507136 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.928027 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236928007, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5279, "file_size": 33507136, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 147971153, "largest_seqno": 148888638, "table_properties": {"data_size": 32159104, "index_size": 1347200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196184, "raw_average_key_size": 24, "raw_value_size": 32159069, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 49841, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5279, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.945233 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5281: 32137 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.945279 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001236945259, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5281, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 135766537, "largest_seqno": 135828684, "table_properties": {"data_size": 32681856, "index_size": 842624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642740, "raw_average_key_size": 20, "raw_value_size": 32681810, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32137, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5281, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.971930 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5282: 49897 keys, 33507008 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.972002 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001236971971, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5282, "file_size": 33507008, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 148888639, "largest_seqno": 149665581, "table_properties": {"data_size": 32156928, "index_size": 1349248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1197528, "raw_average_key_size": 24, "raw_value_size": 32156884, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49897, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5282, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:16.977661 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5283: 32061 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:16.977706 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001236977686, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5283, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 135828685, "largest_seqno": 135909928, "table_properties": {"data_size": 32603456, "index_size": 920960, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641220, "raw_average_key_size": 20, "raw_value_size": 32603445, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32061, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5283, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.010390 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5280: 231460 keys, 33324867 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.010496 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001237010453, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5280, "file_size": 33324867, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139992573, "largest_seqno": 140551362, "table_properties": {"data_size": 27027584, "index_size": 6296448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480880, "raw_average_key_size": 28, "raw_value_size": 27027576, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231460, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5280, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.015821 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5285: 32057 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.015906 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001237015869, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5285, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 135909929, "largest_seqno": 136651412, "table_properties": {"data_size": 32603072, "index_size": 920512, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32603069, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5285, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.020010 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5284: 49860 keys, 33507008 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.020086 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001237020052, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5284, "file_size": 33507008, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 149665582, "largest_seqno": 150453638, "table_properties": {"data_size": 32159424, "index_size": 1346752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1196640, "raw_average_key_size": 24, "raw_value_size": 32159382, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 49860, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001236, "oldest_key_time": 0, "file_creation_time": 1732001236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5284, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.046807 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5288: 20791 keys, 14045504 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.046887 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001237046852, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5288, "file_size": 14045504, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 150453639, "largest_seqno": 150887069, "table_properties": {"data_size": 13401408, "index_size": 643264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 498984, "raw_average_key_size": 24, "raw_value_size": 13401392, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 20791, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5288, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.053674 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-01 2024/11/19-15:27:17.053825 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00246/att-01/compact.done) = No such file or directory: 11.023 sec on http probe failed: 4, retry 2024/11/19-15:27:17.065677 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5287: 32150 keys, 33524224 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.065765 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001237065721, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5287, "file_size": 33524224, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 136651413, "largest_seqno": 136733323, "table_properties": {"data_size": 32696704, "index_size": 826688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32696700, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5287, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.110196 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5290: 32057 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.110250 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001237110228, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5290, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 136733324, "largest_seqno": 136795390, "table_properties": {"data_size": 32602496, "index_size": 921536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602477, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5290, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.149036 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5291: 32059 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.149091 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001237149070, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5291, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 136720244, "largest_seqno": 136820259, "table_properties": {"data_size": 32601920, "index_size": 921792, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32601894, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5291, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.184323 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5286: 231471 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.184379 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001237184357, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5286, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 140405239, "largest_seqno": 141041727, "table_properties": {"data_size": 27024576, "index_size": 6299584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481188, "raw_average_key_size": 28, "raw_value_size": 27024514, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231471, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5286, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.195406 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5292: 32093 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.195452 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001237195432, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5292, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 136747441, "largest_seqno": 136809544, "table_properties": {"data_size": 32638848, "index_size": 885504, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641860, "raw_average_key_size": 20, "raw_value_size": 32638822, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32093, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5292, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.215416 139851179493120 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5289: 344006 keys, 27018179 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.215465 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001237215444, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5289, "file_size": 27018179, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139817829, "largest_seqno": 145384047, "table_properties": {"data_size": 18572800, "index_size": 8444544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6880120, "raw_average_key_size": 20, "raw_value_size": 18572794, "raw_average_value_size": 53, "num_data_blocks": 1, "num_entries": 344006, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5289, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.234971 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5294: 32055 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.235015 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001237234996, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5294, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 136809545, "largest_seqno": 136900511, "table_properties": {"data_size": 32601088, "index_size": 923072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32601033, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5294, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.266816 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-01 2024/11/19-15:27:17.266889 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00247/att-01/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:27:17.284461 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5295: 32057 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.284507 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001237284485, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5295, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 136900512, "largest_seqno": 137608336, "table_properties": {"data_size": 32601024, "index_size": 922880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601010, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5295, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.305728 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5293: 231354 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.305791 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001237305760, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5293, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141000930, "largest_seqno": 141550085, "table_properties": {"data_size": 27029248, "index_size": 6294976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6477912, "raw_average_key_size": 28, "raw_value_size": 27029232, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231354, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5293, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.326541 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5296: 32128 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.326586 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001237326565, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5296, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137608337, "largest_seqno": 137677483, "table_properties": {"data_size": 32674432, "index_size": 849472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642560, "raw_average_key_size": 20, "raw_value_size": 32674381, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32128, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5296, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.448465 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5298: 32058 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.448537 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001237448507, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5298, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137677484, "largest_seqno": 137746424, "table_properties": {"data_size": 32602496, "index_size": 921920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602442, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5298, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.487775 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5297: 231362 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.487832 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001237487810, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5297, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141026252, "largest_seqno": 141637273, "table_properties": {"data_size": 27029696, "index_size": 6294528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6478136, "raw_average_key_size": 28, "raw_value_size": 27029696, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231362, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5297, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.553107 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5299: 32053 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.553164 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001237553142, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5299, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137746425, "largest_seqno": 137788847, "table_properties": {"data_size": 32596672, "index_size": 927168, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32596656, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5299, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.595088 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5301: 32150 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.595144 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001237595122, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5301, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137758482, "largest_seqno": 137819645, "table_properties": {"data_size": 32697600, "index_size": 826304, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32697558, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5301, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.607434 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5300: 231477 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.607489 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001237607467, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5300, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141637274, "largest_seqno": 142277702, "table_properties": {"data_size": 27024064, "index_size": 6300096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481356, "raw_average_key_size": 28, "raw_value_size": 27024018, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231477, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5300, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.711108 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5302: 32054 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.711166 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001237711144, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5302, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137819646, "largest_seqno": 137891646, "table_properties": {"data_size": 32597824, "index_size": 925952, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641080, "raw_average_key_size": 20, "raw_value_size": 32597776, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5302, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.785621 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-02 2024/11/19-15:27:17.785686 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00245/att-02/compact.done) = No such file or directory: 5.971 sec on http probe failed: 2, retry 2024/11/19-15:27:17.812860 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5303: 231452 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:17.812909 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001237812887, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5303, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141999939, "largest_seqno": 142539346, "table_properties": {"data_size": 27026880, "index_size": 6297344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480656, "raw_average_key_size": 28, "raw_value_size": 27026828, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231452, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5303, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:17.955172 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-01 2024/11/19-15:27:17.955262 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00248/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:27:18.016161 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5304: 32055 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.016222 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238016199, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5304, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 137891647, "largest_seqno": 138598974, "table_properties": {"data_size": 32600192, "index_size": 924096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32600152, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5304, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.103643 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5305: 231254 keys, 33325187 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.103704 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001238103681, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5305, "file_size": 33325187, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 142268432, "largest_seqno": 142909394, "table_properties": {"data_size": 27032512, "index_size": 6291840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6475112, "raw_average_key_size": 28, "raw_value_size": 27032476, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231254, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001237, "oldest_key_time": 0, "file_creation_time": 1732001237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5305, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.103764 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5306: 32060 keys, 33524800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.103826 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238103797, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5306, "file_size": 33524800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138598975, "largest_seqno": 138673911, "table_properties": {"data_size": 32607040, "index_size": 916928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32606990, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5306, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.135538 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5308: 32056 keys, 33524800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.135595 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238135573, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5308, "file_size": 33524800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138673912, "largest_seqno": 138767067, "table_properties": {"data_size": 32598336, "index_size": 925632, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32598274, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5308, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.166871 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5309: 32055 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.166941 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238166904, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5309, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138717046, "largest_seqno": 138817063, "table_properties": {"data_size": 32600512, "index_size": 923520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32600461, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5309, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.198548 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5310: 32072 keys, 33524800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.198600 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238198580, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5310, "file_size": 33524800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138719114, "largest_seqno": 138781196, "table_properties": {"data_size": 32617984, "index_size": 905984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641440, "raw_average_key_size": 20, "raw_value_size": 32617926, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32072, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5310, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.226495 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-00 2024/11/19-15:27:18.226602 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00249/att-00/compact.done) = No such file or directory: 11.020 sec on http probe failed: 4, retry 2024/11/19-15:27:18.317997 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5307: 231471 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.318053 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001238318030, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5307, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 142909395, "largest_seqno": 143525479, "table_properties": {"data_size": 27026112, "index_size": 6298112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481188, "raw_average_key_size": 28, "raw_value_size": 27026070, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231471, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5307, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.359591 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5311: 32051 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.359660 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238359629, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5311, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138781197, "largest_seqno": 138833259, "table_properties": {"data_size": 32597376, "index_size": 926528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641020, "raw_average_key_size": 20, "raw_value_size": 32597322, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32051, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5311, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.391021 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5313: 32149 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.391072 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238391051, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5313, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138833260, "largest_seqno": 138911077, "table_properties": {"data_size": 32697024, "index_size": 826880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642980, "raw_average_key_size": 20, "raw_value_size": 32697022, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32149, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5313, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.422582 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5314: 32059 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.422630 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238422610, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5314, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 138911078, "largest_seqno": 139661151, "table_properties": {"data_size": 32603456, "index_size": 920384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603411, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5314, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.429049 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5312: 231464 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.429093 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001238429073, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5312, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 142997716, "largest_seqno": 143560861, "table_properties": {"data_size": 27023552, "index_size": 6300608, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480992, "raw_average_key_size": 28, "raw_value_size": 27023512, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231464, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5312, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.528032 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5315: 32150 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.528086 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238528064, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5315, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139661152, "largest_seqno": 139737362, "table_properties": {"data_size": 32696960, "index_size": 826688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32696905, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5315, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.556690 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5316: 231349 keys, 33325123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.556743 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001238556722, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5316, "file_size": 33325123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 143560862, "largest_seqno": 144131143, "table_properties": {"data_size": 27029888, "index_size": 6294400, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6477772, "raw_average_key_size": 28, "raw_value_size": 27029870, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231349, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5316, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.560068 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-00 2024/11/19-15:27:18.560140 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00250/att-00/compact.done) = No such file or directory: 7.006 sec on http probe failed: 2, retry 2024/11/19-15:27:18.562210 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5317: 32059 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.562253 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238562232, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5317, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139737363, "largest_seqno": 139789428, "table_properties": {"data_size": 32602496, "index_size": 921408, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32602489, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5317, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.594385 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5319: 32060 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.594431 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238594411, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5319, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139724812, "largest_seqno": 139794820, "table_properties": {"data_size": 32604160, "index_size": 920064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32604153, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5319, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.636768 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5320: 32095 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.636823 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238636801, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5320, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139781488, "largest_seqno": 139832593, "table_properties": {"data_size": 32640064, "index_size": 883712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641900, "raw_average_key_size": 20, "raw_value_size": 32640009, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32095, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5320, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.686300 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5321: 32056 keys, 33524352 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.686355 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238686332, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5321, "file_size": 33524352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139832594, "largest_seqno": 139934670, "table_properties": {"data_size": 32602304, "index_size": 921216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32602283, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5321, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.686400 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5318: 231336 keys, 33325123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.686444 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001238686424, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5318, "file_size": 33325123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 143994691, "largest_seqno": 144534876, "table_properties": {"data_size": 27029760, "index_size": 6294528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6477408, "raw_average_key_size": 28, "raw_value_size": 27029732, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231336, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5318, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.759197 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5322: 32056 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.759271 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238759239, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5322, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139934671, "largest_seqno": 140645643, "table_properties": {"data_size": 32602816, "index_size": 920896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32602810, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5322, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.817910 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5324: 32129 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.817973 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238817952, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5324, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 140645644, "largest_seqno": 140686780, "table_properties": {"data_size": 32676928, "index_size": 846656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642580, "raw_average_key_size": 20, "raw_value_size": 32676927, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32129, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5324, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.818013 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5323: 231457 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.818058 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001238818038, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5323, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 144171719, "largest_seqno": 144822937, "table_properties": {"data_size": 27025216, "index_size": 6298944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480796, "raw_average_key_size": 28, "raw_value_size": 27025178, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231457, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5323, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.866839 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5325: 32058 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.866894 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238866871, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5325, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 140686781, "largest_seqno": 140759803, "table_properties": {"data_size": 32602752, "index_size": 921024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602690, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5325, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:18.903930 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5327: 32058 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:18.904000 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001238903968, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5327, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 140742740, "largest_seqno": 140782744, "table_properties": {"data_size": 32603584, "index_size": 920640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603529, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5327, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:19.030002 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5326: 231459 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:19.030060 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001239030038, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5326, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 144822938, "largest_seqno": 145422503, "table_properties": {"data_size": 27026304, "index_size": 6297920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480852, "raw_average_key_size": 28, "raw_value_size": 27026246, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231459, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5326, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:19.055216 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5328: 32151 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:19.055279 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001239055249, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5328, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 140771863, "largest_seqno": 140854179, "table_properties": {"data_size": 32696512, "index_size": 827072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32696490, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001238, "oldest_key_time": 0, "file_creation_time": 1732001238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5328, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:19.061925 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-01 2024/11/19-15:27:19.062046 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00246/att-01/compact.done) = No such file or directory: 13.033 sec on http probe failed: 5, giveup 2024/11/19-15:27:19.062073 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00246/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 222.689 sec, real 13.359 (submit 317.653 ms wait 13.033 s), input 222.689 MiB, send shutdown 2024/11/19-15:27:19.070770 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-01 2024/11/19-15:27:19.071122 139851169007360 [dcompact_etcd.cc:1773] job-00246/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:19.087061 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5330: 32054 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:19.087105 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001239087085, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5330, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 140854180, "largest_seqno": 140926401, "table_properties": {"data_size": 32601728, "index_size": 922368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641080, "raw_average_key_size": 20, "raw_value_size": 32601697, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001239, "oldest_key_time": 0, "file_creation_time": 1732001239, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5330, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:19.119752 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5331: 32055 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:19.119814 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001239119783, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5331, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 140926402, "largest_seqno": 141600310, "table_properties": {"data_size": 32600320, "index_size": 923712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32600295, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001239, "oldest_key_time": 0, "file_creation_time": 1732001239, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5331, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:19.147039 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5329: 231258 keys, 33325187 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:19.147088 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001239147067, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5329, "file_size": 33325187, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 144993229, "largest_seqno": 145531904, "table_properties": {"data_size": 27032832, "index_size": 6291520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6475224, "raw_average_key_size": 28, "raw_value_size": 27032812, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231258, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001239, "oldest_key_time": 0, "file_creation_time": 1732001239, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5329, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:19.167173 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5332: 32073 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:19.167218 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001239167197, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5332, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141600311, "largest_seqno": 141678466, "table_properties": {"data_size": 32617600, "index_size": 905984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641460, "raw_average_key_size": 20, "raw_value_size": 32617539, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32073, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001239, "oldest_key_time": 0, "file_creation_time": 1732001239, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5332, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:19.258662 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5334: 32053 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:19.258717 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001239258695, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5334, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141678467, "largest_seqno": 141761780, "table_properties": {"data_size": 32598720, "index_size": 924928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32598678, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001239, "oldest_key_time": 0, "file_creation_time": 1732001239, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5334, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:19.259004 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5333: 231426 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:19.259049 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001239259029, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5333, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 145413222, "largest_seqno": 146034809, "table_properties": {"data_size": 27026304, "index_size": 6297920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6479928, "raw_average_key_size": 28, "raw_value_size": 27026292, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231426, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001239, "oldest_key_time": 0, "file_creation_time": 1732001239, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5333, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:19.275039 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-01 2024/11/19-15:27:19.275114 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00247/att-01/compact.done) = No such file or directory: 9.015 sec on http probe failed: 3, retry 2024/11/19-15:27:19.305979 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5335: 32055 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:19.306026 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001239306005, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5335, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141722270, "largest_seqno": 141802280, "table_properties": {"data_size": 32600960, "index_size": 923072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32600903, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001239, "oldest_key_time": 0, "file_creation_time": 1732001239, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5335, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:19.353504 139851169007360 [dcompact_etcd.cc:1163] job-00246/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.282 sec, input = 222.689 MiB 2024/11/19-15:27:19.417135 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5337: 32092 keys, 33524352 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:19.417187 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001239417165, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5337, "file_size": 33524352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141743834, "largest_seqno": 141811938, "table_properties": {"data_size": 32637120, "index_size": 886400, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641840, "raw_average_key_size": 20, "raw_value_size": 32637115, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32092, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001239, "oldest_key_time": 0, "file_creation_time": 1732001239, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5337, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:19.417254 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5336: 231454 keys, 33324931 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:19.417295 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001239417276, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5336, "file_size": 33324931, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 145989464, "largest_seqno": 146497985, "table_properties": {"data_size": 27025152, "index_size": 6298944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6480712, "raw_average_key_size": 28, "raw_value_size": 27025148, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231454, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001239, "oldest_key_time": 0, "file_creation_time": 1732001239, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5336, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:19.448718 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5338: 32053 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:19.448764 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001239448743, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5338, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141811939, "largest_seqno": 141886906, "table_properties": {"data_size": 32600768, "index_size": 923648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32600738, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001239, "oldest_key_time": 0, "file_creation_time": 1732001239, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5338, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:19.569461 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5340: 32055 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:19.569516 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001239569495, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5340, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141886907, "largest_seqno": 142557014, "table_properties": {"data_size": 32601408, "index_size": 923008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32601389, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001239, "oldest_key_time": 0, "file_creation_time": 1732001239, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5340, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:19.773881 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5339: 231415 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:19.773947 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001239773916, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5339, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 146075188, "largest_seqno": 146646495, "table_properties": {"data_size": 27027520, "index_size": 6296704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6479620, "raw_average_key_size": 28, "raw_value_size": 27027470, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231415, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001239, "oldest_key_time": 0, "file_creation_time": 1732001239, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5339, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:19.795432 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-02 2024/11/19-15:27:19.795505 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00245/att-02/compact.done) = No such file or directory: 7.980 sec on http probe failed: 3, retry 2024/11/19-15:27:19.960070 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-01 2024/11/19-15:27:19.960156 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00248/att-01/compact.done) = No such file or directory: 7.007 sec on http probe failed: 2, retry 2024/11/19-15:27:20.080624 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5341: 32123 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.080680 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240080658, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5341, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 142557015, "largest_seqno": 142660103, "table_properties": {"data_size": 32670592, "index_size": 853824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642460, "raw_average_key_size": 20, "raw_value_size": 32670557, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32123, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001239, "oldest_key_time": 0, "file_creation_time": 1732001239, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5341, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.183030 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5342: 231298 keys, 33325123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.183087 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001240183065, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5342, "file_size": 33325123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 146646496, "largest_seqno": 147245743, "table_properties": {"data_size": 27031872, "index_size": 6292416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6476344, "raw_average_key_size": 28, "raw_value_size": 27031820, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231298, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001239, "oldest_key_time": 0, "file_creation_time": 1732001239, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5342, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.231848 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-00 2024/11/19-15:27:20.231952 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00249/att-00/compact.done) = No such file or directory: 13.030 sec on http probe failed: 5, giveup 2024/11/19-15:27:20.231978 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00249/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.643 sec, real 13.349 (submit 313.690 ms wait 13.030 s), input 253.643 MiB, send shutdown 2024/11/19-15:27:20.240090 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-00 2024/11/19-15:27:20.273335 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5343: 32057 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.273401 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240273370, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5343, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 142660104, "largest_seqno": 142751859, "table_properties": {"data_size": 32602176, "index_size": 922048, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602146, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5343, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.273402 139851148035840 [dcompact_etcd.cc:1773] job-00249/att-00: CleanFiles 0.033 sec : 0.000 2024/11/19-15:27:20.294313 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5344: 231479 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.294364 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001240294341, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5344, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 146987433, "largest_seqno": 147496740, "table_properties": {"data_size": 27026048, "index_size": 6298176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481412, "raw_average_key_size": 28, "raw_value_size": 27025986, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231479, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5344, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.313501 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5345: 32052 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.313552 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240313529, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5345, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 142719914, "largest_seqno": 142799924, "table_properties": {"data_size": 32596864, "index_size": 927232, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641040, "raw_average_key_size": 20, "raw_value_size": 32596817, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32052, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5345, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.345409 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5347: 32152 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.345458 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240345437, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5347, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 142723907, "largest_seqno": 142786067, "table_properties": {"data_size": 32697408, "index_size": 826688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643040, "raw_average_key_size": 20, "raw_value_size": 32697358, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32152, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5347, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.377672 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5348: 32055 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.377722 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240377701, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5348, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 142786068, "largest_seqno": 142868003, "table_properties": {"data_size": 32598336, "index_size": 926016, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32598330, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5348, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.398896 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5346: 231498 keys, 33324931 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.398950 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001240398930, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5346, "file_size": 33324931, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 147347192, "largest_seqno": 147938621, "table_properties": {"data_size": 27027520, "index_size": 6296576, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481944, "raw_average_key_size": 28, "raw_value_size": 27027460, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231498, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5346, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.409708 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5349: 32054 keys, 33527232 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.409752 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240409732, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5349, "file_size": 33527232, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 142868004, "largest_seqno": 143547180, "table_properties": {"data_size": 32599168, "index_size": 927232, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641080, "raw_average_key_size": 20, "raw_value_size": 32599148, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5349, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.441481 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5351: 32149 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.441526 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240441506, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5351, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 143547181, "largest_seqno": 143656054, "table_properties": {"data_size": 32697344, "index_size": 826368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642980, "raw_average_key_size": 20, "raw_value_size": 32697287, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32149, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5351, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.473659 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5352: 32150 keys, 33524224 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.473713 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240473691, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5352, "file_size": 33524224, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 143656055, "largest_seqno": 143732021, "table_properties": {"data_size": 32696576, "index_size": 826816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32696562, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5352, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.506174 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5353: 32057 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.506227 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240506205, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5353, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 143732022, "largest_seqno": 143794087, "table_properties": {"data_size": 32602688, "index_size": 921536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602673, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5353, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.518035 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5350: 231287 keys, 33325123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.518086 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001240518064, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5350, "file_size": 33325123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 147938622, "largest_seqno": 148487336, "table_properties": {"data_size": 27031552, "index_size": 6292736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6476036, "raw_average_key_size": 28, "raw_value_size": 27031550, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231287, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5350, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.539821 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5354: 32058 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.539883 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240539849, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5354, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 143717467, "largest_seqno": 143797477, "table_properties": {"data_size": 32603456, "index_size": 920128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603432, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5354, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.569082 139851148035840 [dcompact_etcd.cc:1163] job-00249/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.296 sec, input = 253.643 MiB 2024/11/19-15:27:20.569953 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-00 2024/11/19-15:27:20.570002 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00250/att-00/compact.done) = No such file or directory: 9.014 sec on http probe failed: 3, retry 2024/11/19-15:27:20.572000 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5356: 32150 keys, 33524288 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.572045 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240572023, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5356, "file_size": 33524288, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 143766141, "largest_seqno": 143857868, "table_properties": {"data_size": 32697088, "index_size": 826368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643000, "raw_average_key_size": 20, "raw_value_size": 32697048, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5356, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.604155 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5357: 32058 keys, 33524352 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.604205 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240604184, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5357, "file_size": 33524352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 143857869, "largest_seqno": 143938703, "table_properties": {"data_size": 32602304, "index_size": 921216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602279, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5357, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.636773 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5355: 231410 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.636846 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001240636815, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5355, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 148016054, "largest_seqno": 148579942, "table_properties": {"data_size": 27028288, "index_size": 6295872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6479480, "raw_average_key_size": 28, "raw_value_size": 27028240, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231410, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5355, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.636852 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5358: 32059 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.636907 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240636887, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5358, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 143938704, "largest_seqno": 144609735, "table_properties": {"data_size": 32602752, "index_size": 920832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32602700, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5358, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.668768 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5360: 32138 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.668825 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240668802, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5360, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 144609736, "largest_seqno": 144681598, "table_properties": {"data_size": 32685504, "index_size": 838592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642760, "raw_average_key_size": 20, "raw_value_size": 32685474, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32138, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5360, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.700434 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5361: 32058 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.700488 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240700465, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5361, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 144681599, "largest_seqno": 144764684, "table_properties": {"data_size": 32602944, "index_size": 920896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602882, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5361, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.732971 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5362: 32058 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.733025 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240733003, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5362, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 144725547, "largest_seqno": 144785556, "table_properties": {"data_size": 32603712, "index_size": 920512, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603663, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5362, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.755609 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5359: 231473 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.755664 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001240755642, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5359, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 148579943, "largest_seqno": 149149721, "table_properties": {"data_size": 27025280, "index_size": 6298880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481244, "raw_average_key_size": 28, "raw_value_size": 27025266, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231473, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5359, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.775959 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5363: 32151 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.776009 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240775989, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5363, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 144766744, "largest_seqno": 144848700, "table_properties": {"data_size": 32697728, "index_size": 826624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32697713, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5363, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.838017 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5365: 32058 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.838071 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240838049, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5365, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 144848701, "largest_seqno": 144926607, "table_properties": {"data_size": 32602368, "index_size": 921536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602332, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5365, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.874958 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5364: 231424 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.875012 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001240874990, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5364, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 149014042, "largest_seqno": 149492534, "table_properties": {"data_size": 27028352, "index_size": 6295872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6479872, "raw_average_key_size": 28, "raw_value_size": 27028320, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231424, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5364, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.885831 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5366: 32057 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.885883 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240885861, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5366, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 144926608, "largest_seqno": 145590082, "table_properties": {"data_size": 32602688, "index_size": 921216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602636, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5366, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.932828 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5368: 32098 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.932894 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240932871, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5368, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 145590083, "largest_seqno": 145670647, "table_properties": {"data_size": 32642688, "index_size": 881088, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641960, "raw_average_key_size": 20, "raw_value_size": 32642643, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32098, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5368, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:20.987757 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5369: 32055 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:20.987812 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001240987789, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5369, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 145670648, "largest_seqno": 145763378, "table_properties": {"data_size": 32601600, "index_size": 922816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32601587, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5369, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.025977 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5367: 231296 keys, 33325123 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.026031 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001241026009, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5367, "file_size": 33325123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 149201316, "largest_seqno": 149792064, "table_properties": {"data_size": 27031872, "index_size": 6292416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6476288, "raw_average_key_size": 28, "raw_value_size": 27031872, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231296, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5367, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.026827 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5370: 32056 keys, 33524992 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.026879 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001241026853, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5370, "file_size": 33524992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 145711763, "largest_seqno": 145791773, "table_properties": {"data_size": 32601344, "index_size": 922816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32601335, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001240, "oldest_key_time": 0, "file_creation_time": 1732001240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5370, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.142564 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5372: 32134 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.142623 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001241142599, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5372, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 145735430, "largest_seqno": 145803576, "table_properties": {"data_size": 32680448, "index_size": 843200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642680, "raw_average_key_size": 20, "raw_value_size": 32680436, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32134, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5372, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.160003 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5371: 231479 keys, 33324995 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.160072 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001241160041, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5371, "file_size": 33324995, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 149792065, "largest_seqno": 150392213, "table_properties": {"data_size": 27026688, "index_size": 6297472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481412, "raw_average_key_size": 28, "raw_value_size": 27026642, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231479, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5371, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.180638 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5373: 32056 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.180696 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001241180667, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5373, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 145803577, "largest_seqno": 145878465, "table_properties": {"data_size": 32602624, "index_size": 921856, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32602571, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5373, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.281139 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-01 2024/11/19-15:27:21.281221 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00247/att-01/compact.done) = No such file or directory: 11.024 sec on http probe failed: 4, retry 2024/11/19-15:27:21.297370 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5375: 32053 keys, 33524544 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.297419 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001241297398, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5375, "file_size": 33524544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 145878466, "largest_seqno": 146545886, "table_properties": {"data_size": 32596992, "index_size": 926720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32596976, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5375, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.301000 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5374: 231482 keys, 33325059 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.301043 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001241301022, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5374, "file_size": 33325059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 149982114, "largest_seqno": 150493564, "table_properties": {"data_size": 27026752, "index_size": 6297472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6481496, "raw_average_key_size": 28, "raw_value_size": 27026736, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 231482, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5374, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.329021 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5376: 32149 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.329065 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001241329044, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5376, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 146545887, "largest_seqno": 146657599, "table_properties": {"data_size": 32696960, "index_size": 826944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642980, "raw_average_key_size": 20, "raw_value_size": 32696936, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32149, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5376, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.403763 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5378: 32055 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.403818 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001241403796, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5378, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 146657600, "largest_seqno": 146733837, "table_properties": {"data_size": 32598848, "index_size": 925632, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32598827, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5378, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.403980 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5377: 180036 keys, 25957763 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.404025 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001241404005, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5377, "file_size": 25957763, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 150493565, "largest_seqno": 150873164, "table_properties": {"data_size": 21028224, "index_size": 4928704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 5041008, "raw_average_key_size": 28, "raw_value_size": 21028176, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 180036, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5377, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.467729 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5380: 32057 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.467783 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001241467762, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5380, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 133737040, "largest_seqno": 133799105, "table_properties": {"data_size": 32601408, "index_size": 922944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601386, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5380, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.467956 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5379: 32055 keys, 33524928 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.468016 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001241467988, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5379, "file_size": 33524928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 146719725, "largest_seqno": 146779732, "table_properties": {"data_size": 32600448, "index_size": 923648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641100, "raw_average_key_size": 20, "raw_value_size": 32600387, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5379, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.588100 139849738749696 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5381: 24416 keys, 25571392 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.588161 139849738749696 EVENT_LOG_v1 {"time_micros": 1732001241588134, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5381, "file_size": 25571392, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 133725828, "largest_seqno": 133825841, "table_properties": {"data_size": 24829184, "index_size": 741376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 488320, "raw_average_key_size": 20, "raw_value_size": 24829146, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 24416, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5381, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.610864 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5382: 32075 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.610911 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001241610890, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5382, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 146725891, "largest_seqno": 146790977, "table_properties": {"data_size": 32620224, "index_size": 904064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641500, "raw_average_key_size": 20, "raw_value_size": 32620193, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32075, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5382, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.641895 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5383: 32053 keys, 33525056 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.641950 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001241641929, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5383, "file_size": 33525056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 146790978, "largest_seqno": 146899791, "table_properties": {"data_size": 32599424, "index_size": 924800, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641060, "raw_average_key_size": 20, "raw_value_size": 32599364, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5383, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.672421 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5384: 32056 keys, 33524416 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.672469 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001241672448, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5384, "file_size": 33524416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 146899792, "largest_seqno": 147536837, "table_properties": {"data_size": 32600768, "index_size": 922816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641120, "raw_average_key_size": 20, "raw_value_size": 32600712, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32056, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5384, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.703092 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5385: 32095 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.703141 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001241703120, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5385, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 147536838, "largest_seqno": 147608977, "table_properties": {"data_size": 32642496, "index_size": 881920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641900, "raw_average_key_size": 20, "raw_value_size": 32642443, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32095, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5385, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.746231 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5386: 32057 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.746283 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001241746262, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5386, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 147608978, "largest_seqno": 147674906, "table_properties": {"data_size": 32599744, "index_size": 924032, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32599689, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5386, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.801096 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-02 2024/11/19-15:27:21.801172 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00245/att-02/compact.done) = No such file or directory: 9.990 sec on http probe failed: 4, retry 2024/11/19-15:27:21.837153 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5387: 32151 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.837203 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001241837181, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5387, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 147674907, "largest_seqno": 147746922, "table_properties": {"data_size": 32697728, "index_size": 826624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32697724, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5387, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.869160 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5388: 32058 keys, 33524480 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.869210 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001241869188, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5388, "file_size": 33524480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 147746923, "largest_seqno": 147818943, "table_properties": {"data_size": 32603520, "index_size": 920128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32603483, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5388, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.900110 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5389: 32151 keys, 33524800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.900158 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001241900137, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5389, "file_size": 33524800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 147818944, "largest_seqno": 147891058, "table_properties": {"data_size": 32697536, "index_size": 826432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32697499, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5389, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.934773 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5390: 32057 keys, 33525312 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:21.934840 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001241934807, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5390, "file_size": 33525312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 147891059, "largest_seqno": 147932125, "table_properties": {"data_size": 32603328, "index_size": 921152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32603276, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5390, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:21.969635 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-01 2024/11/19-15:27:21.969728 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00248/att-01/compact.done) = No such file or directory: 9.012 sec on http probe failed: 3, retry 2024/11/19-15:27:22.034080 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5391: 32059 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.034125 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242034104, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5391, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 147932126, "largest_seqno": 148562080, "table_properties": {"data_size": 32603072, "index_size": 920832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603041, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001241, "oldest_key_time": 0, "file_creation_time": 1732001241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5391, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.064911 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5392: 32136 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.064963 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242064943, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5392, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 148562081, "largest_seqno": 148634010, "table_properties": {"data_size": 32681984, "index_size": 841920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642720, "raw_average_key_size": 20, "raw_value_size": 32681921, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32136, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5392, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.095895 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5393: 32059 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.095954 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242095932, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5393, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 148634011, "largest_seqno": 148706112, "table_properties": {"data_size": 32603520, "index_size": 920896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603485, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5393, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.133818 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5394: 32060 keys, 33525120 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.133878 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242133856, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5394, "file_size": 33525120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 148706113, "largest_seqno": 148809247, "table_properties": {"data_size": 32603776, "index_size": 920512, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641200, "raw_average_key_size": 20, "raw_value_size": 32603754, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32060, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5394, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.235466 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5395: 32151 keys, 33524608 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.235520 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242235498, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5395, "file_size": 33524608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 148809248, "largest_seqno": 148881569, "table_properties": {"data_size": 32697088, "index_size": 826688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32697042, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5395, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.266415 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5396: 32057 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.266467 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242266445, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5396, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 148881570, "largest_seqno": 148922634, "table_properties": {"data_size": 32602304, "index_size": 921536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32602302, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5396, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.297255 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5397: 32059 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.297305 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242297283, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5397, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 148922635, "largest_seqno": 149552720, "table_properties": {"data_size": 32603904, "index_size": 920128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32603844, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5397, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.345744 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5398: 32096 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.345793 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242345772, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5398, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 149552721, "largest_seqno": 149655827, "table_properties": {"data_size": 32640256, "index_size": 883776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641920, "raw_average_key_size": 20, "raw_value_size": 32640221, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32096, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5398, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.387956 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5399: 32059 keys, 33524736 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.388005 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242387983, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5399, "file_size": 33524736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 149655828, "largest_seqno": 149726972, "table_properties": {"data_size": 32602496, "index_size": 921408, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641180, "raw_average_key_size": 20, "raw_value_size": 32602453, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5399, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.442900 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5400: 32057 keys, 33524864 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.442960 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242442939, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5400, "file_size": 33524864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 149726973, "largest_seqno": 149768942, "table_properties": {"data_size": 32601280, "index_size": 922752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32601239, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5400, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.485021 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5401: 32135 keys, 33524672 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.485074 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242485052, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5401, "file_size": 33524672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 149768943, "largest_seqno": 149838159, "table_properties": {"data_size": 32680576, "index_size": 843264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 642700, "raw_average_key_size": 20, "raw_value_size": 32680576, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32135, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5401, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.579077 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-00 2024/11/19-15:27:22.579150 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00250/att-00/compact.done) = No such file or directory: 11.024 sec on http probe failed: 4, retry 2024/11/19-15:27:22.586093 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5402: 32058 keys, 33525248 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.586138 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242586117, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5402, "file_size": 33525248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 149838160, "largest_seqno": 149910147, "table_properties": {"data_size": 32602560, "index_size": 921856, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641160, "raw_average_key_size": 20, "raw_value_size": 32602538, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5402, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.616821 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5403: 32051 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.616879 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242616859, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5403, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 149910148, "largest_seqno": 150545370, "table_properties": {"data_size": 32597760, "index_size": 926592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641020, "raw_average_key_size": 20, "raw_value_size": 32597752, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32051, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5403, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.653120 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5404: 32151 keys, 33524800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.653170 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242653149, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5404, "file_size": 33524800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 150545371, "largest_seqno": 150614391, "table_properties": {"data_size": 32697024, "index_size": 826944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 643020, "raw_average_key_size": 20, "raw_value_size": 32696981, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5404, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.766133 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5405: 32051 keys, 33524800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.766186 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242766165, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5405, "file_size": 33524800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 150614392, "largest_seqno": 150683634, "table_properties": {"data_size": 32598528, "index_size": 925440, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641020, "raw_average_key_size": 20, "raw_value_size": 32598494, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 32051, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5405, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.825886 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5406: 32057 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.825946 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242825925, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5406, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 150683635, "largest_seqno": 150761682, "table_properties": {"data_size": 32600576, "index_size": 923776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641140, "raw_average_key_size": 20, "raw_value_size": 32600521, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5406, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.864703 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5407: 32077 keys, 33525184 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.864751 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242864731, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5407, "file_size": 33525184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 150761683, "largest_seqno": 150833604, "table_properties": {"data_size": 32621248, "index_size": 903104, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 641540, "raw_average_key_size": 20, "raw_value_size": 32621226, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 32077, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5407, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.895281 139849728263936 [compaction_job.cc:1948] [default] [JOB 252] Generated table #5408: 29708 keys, 31081920 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:22.895356 139849728263936 EVENT_LOG_v1 {"time_micros": 1732001242895320, "cf_name": "default", "job": 252, "event": "table_file_creation", "file_number": 5408, "file_size": 31081920, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 150833605, "largest_seqno": 150897072, "table_properties": {"data_size": 30212864, "index_size": 868224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 594160, "raw_average_key_size": 20, "raw_value_size": 30212801, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 29708, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001242, "oldest_key_time": 0, "file_creation_time": 1732001242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5408, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...0 records...]"}} 2024/11/19-15:27:22.895506 139851179493120 [compaction_job.cc:703] job-00252: subcompact[0], size: 0.476699 G, files: /storage/mytopling/.rocksdb [005250,005253,005254,005256,005258,005261,005264,005267,005269,005271,005273,005276,005279,005282,005284,005288,005289] 2024/11/19-15:27:22.895538 139851179493120 [compaction_job.cc:703] job-00252: subcompact[1], size: 1.078109 G, files: /storage/mytopling/.rocksdb [005251,005260,005266,005274,005280,005286,005293,005297,005300,005303,005305,005307,005312,005316,005318,005323,005326,005329,005333,005336,005339,005342,005344,005346,005350,005355,005359,005364,005367,005371,005374,005377,005380,005381] 2024/11/19-15:27:22.895583 139851179493120 [compaction_job.cc:703] job-00252: subcompact[2], size: 3.618240 G, files: /storage/mytopling/.rocksdb [005252,005255,005257,005259,005262,005263,005265,005268,005270,005272,005275,005277,005278,005281,005283,005285,005287,005290,005291,005292,005294,005295,005296,005298,005299,005301,005302,005304,005306,005308,005309,005310,005311,005313,005314,005315,005317,005319,005320,005321,005322,005324,005325,005327,005328,005330,005331,005332,005334,005335,005337,005338,005340,005341,005343,005345,005347,005348,005349,005351,005352,005353,005354,005356,005357,005358,005360,005361,005362,005363,005365,005366,005368,005369,005370,005372,005373,005375,005376,005378,005379,005382,005383,005384,005385,005386,005387,005388,005389,005390,005391,005392,005393,005394,005395,005396,005397,005398,005399,005400,005401,005402,005403,005404,005405,005406,005407,005408] 2024/11/19-15:27:23.062797 139851179493120 (Original Log Time 2024/11/19-15:27:16.321472) EVENT_LOG_v1 {"time_micros": 1732001236321461, "job": 252, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [5247, 5232, 5219, 5192, 5188, 5173, 5168, 5059], "files_L1": [5105, 5107, 5109, 5110, 5111, 5112, 5114, 5115, 5116, 5117, 5119, 5121, 5122, 5124, 5125, 5126, 5127, 5129, 5130, 5131, 5132, 5134, 5135, 5136, 5138, 5139, 5140, 5141, 5143, 5144, 5145, 5147, 5149, 5151, 5152, 5153, 5154, 5155, 5156, 5157, 5158], "score": 3.18693, "input_data_size": 5682608015, "oldest_snapshot_seqno": 150953291} 2024/11/19-15:27:23.062812 139851179493120 (Original Log Time 2024/11/19-15:27:23.061685) [compaction_job.cc:2022] [default] [JOB 252] Compacted 8@0 + 41@1 files to L1 => 5173047449 bytes 2024/11/19-15:27:23.062832 139851179493120 (Original Log Time 2024/11/19-15:27:23.062394) [compaction_job.cc:1224] [default] compacted to: files[0 159 74 70 0 0 0] max score 9.65 (209 files need compaction), MB/sec: 864.4 rd, 786.9 wr, level 1, files in(8, 41) out(159 +0 blob) MB in(4125.0, 1294.4 +0.0 blob) out(4933.4 +0.0 blob), read-write-amplify(2.5) write-amplify(1.2) OK, records in: 12379172, records dropped: 0 output_compression: Snappy 2024/11/19-15:27:23.062838 139851179493120 (Original Log Time 2024/11/19-15:27:23.062463) EVENT_LOG_v1 {"time_micros": 1732001243062410, "cf": "default", "job": 252, "event": "compaction_finished", "compaction_time_micros": 6574117, "compaction_time_cpu_micros": 8007346, "output_level": 1, "num_output_files": 159, "total_output_size": 5173047449, "num_input_records": 12379172, "num_output_records": 12379172, "num_subcompactions": 3, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 159, 74, 70, 0, 0, 0]} 2024/11/19-15:27:23.062933 139851179493120 [compaction_job.cc:2437] [default] [JOB 253] Compacting 1@1 files to L2, score 9.65, subcompactions 1 : 1 2024/11/19-15:27:23.062943 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 251 Base level 1, inputs: [5250(7923B)] 2024/11/19-15:27:23.289424 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-01 2024/11/19-15:27:23.289534 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00247/att-01/compact.done) = No such file or directory: 13.030 sec on http probe failed: 5, giveup 2024/11/19-15:27:23.289561 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00247/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.893 sec, real 13.345 (submit 306.908 ms wait 13.030 s), input 253.893 MiB, send shutdown 2024/11/19-15:27:23.298414 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-01 2024/11/19-15:27:23.298681 139851137550080 [dcompact_etcd.cc:1773] job-00247/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:23.454626 139851179493120 [dcompact_etcd.cc:1163] job-00253/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.391 sec, input = 3.013 KiB 2024/11/19-15:27:23.463340 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-00 2024/11/19-15:27:23.463407 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00253/att-00/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:27:23.473139 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-00 2024/11/19-15:27:23.473192 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00253/att-00/compact.done) = No such file or directory: 0.009 sec on http probe failed: 2, retry 2024/11/19-15:27:23.478143 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-00 2024/11/19-15:27:23.478189 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00253/att-00/compact.done) = No such file or directory: 0.019 sec on http probe failed: 3, retry 2024/11/19-15:27:23.487456 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-00 2024/11/19-15:27:23.487505 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00253/att-00/compact.done) = No such file or directory: 0.024 sec on http probe failed: 4, retry 2024/11/19-15:27:23.495412 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-00 2024/11/19-15:27:23.495459 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00253/att-00/compact.done) = No such file or directory: 0.033 sec on http probe failed: 5, giveup 2024/11/19-15:27:23.495473 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00253/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.432 (submit 391.186 ms wait 0.033 s), input 3.013 KiB, send shutdown 2024/11/19-15:27:23.499756 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-00 2024/11/19-15:27:23.512602 139851179493120 [dcompact_etcd.cc:1773] job-00253/att-00: CleanFiles 0.013 sec : 0.000 2024/11/19-15:27:23.602495 139851137550080 [dcompact_etcd.cc:1163] job-00247/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.304 sec, input = 253.893 MiB 2024/11/19-15:27:23.783877 139851179493120 [dcompact_etcd.cc:1163] job-00253/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.271 sec, input = 3.013 KiB 2024/11/19-15:27:23.792282 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-01 2024/11/19-15:27:23.792347 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00253/att-01/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:27:23.801706 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-01 2024/11/19-15:27:23.801755 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00253/att-01/compact.done) = No such file or directory: 0.009 sec on http probe failed: 2, retry 2024/11/19-15:27:23.806312 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-02 2024/11/19-15:27:23.806365 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00245/att-02/compact.done) = No such file or directory: 11.995 sec on http probe failed: 5, giveup 2024/11/19-15:27:23.806381 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00245/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 158.601 sec, real 12.265 (submit 264.099 ms wait 11.995 s), input 158.601 MiB, send shutdown 2024/11/19-15:27:23.809391 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-01 2024/11/19-15:27:23.809437 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00253/att-01/compact.done) = No such file or directory: 0.018 sec on http probe failed: 3, retry 2024/11/19-15:27:23.811764 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00245/att-02 2024/11/19-15:27:23.811982 139851127064320 [dcompact_etcd.cc:1773] job-00245/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:23.812210 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00245) = success 2024/11/19-15:27:23.812216 139851127064320 [dcompact_etcd.cc:1773] job-00245/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:23.812233 139851127064320 (Original Log Time 2024/11/19-15:26:46.884498) EVENT_LOG_v1 {"time_micros": 1732001206884492, "job": 245, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4631, 4617, 4787, 4619], "score": 2.74847, "input_data_size": 163722082, "oldest_snapshot_seqno": 149883333} 2024/11/19-15:27:23.812240 139851127064320 [compaction_job.cc:2437] [default] [JOB 245] Compacting 4@2 files to L3, score 2.75, subcompactions 1 : 1 2024/11/19-15:27:23.812248 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 243 Base level 2, inputs: [4631(31MB) 4617(31MB) 4787(62MB) 4619(31MB)] 2024/11/19-15:27:23.812464 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005409.sst) /storage/mytopling/..//tmp/Topling-82038-qtjmhb 2024/11/19-15:27:23.816808 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-01 2024/11/19-15:27:23.816858 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00253/att-01/compact.done) = No such file or directory: 0.026 sec on http probe failed: 4, retry 2024/11/19-15:27:23.822079 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-01 2024/11/19-15:27:23.822129 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00253/att-01/compact.done) = No such file or directory: 0.033 sec on http probe failed: 5, giveup 2024/11/19-15:27:23.822142 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00253/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.309 (submit 271.174 ms wait 0.033 s), input 3.013 KiB, send shutdown 2024/11/19-15:27:23.827865 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-01 2024/11/19-15:27:23.828184 139851179493120 [dcompact_etcd.cc:1773] job-00253/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:23.974583 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-01 2024/11/19-15:27:23.974694 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00248/att-01/compact.done) = No such file or directory: 11.022 sec on http probe failed: 4, retry 2024/11/19-15:27:24.034675 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005409.sst): first pass time = 0.22's, 604.044'MB/sec 2024/11/19-15:27:24.034678 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005409.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:27:24.034706 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005409.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0083 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:27:24.041143 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005409.sst): index pass time = 0.01's, 160.905'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 129429 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:27:24.097576 139851179493120 [dcompact_etcd.cc:1163] job-00253/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.269 sec, input = 3.013 KiB 2024/11/19-15:27:24.105562 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-02 2024/11/19-15:27:24.105627 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00253/att-02/compact.done) = No such file or directory: 0.000 sec on http probe failed: 1, retry 2024/11/19-15:27:24.110317 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-02 2024/11/19-15:27:24.110366 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00253/att-02/compact.done) = No such file or directory: 0.008 sec on http probe failed: 2, retry 2024/11/19-15:27:24.118848 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-02 2024/11/19-15:27:24.118933 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00253/att-02/compact.done) = No such file or directory: 0.013 sec on http probe failed: 3, retry 2024/11/19-15:27:24.128302 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-02 2024/11/19-15:27:24.128360 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00253/att-02/compact.done) = No such file or directory: 0.021 sec on http probe failed: 4, retry 2024/11/19-15:27:24.132853 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-02 2024/11/19-15:27:24.132910 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00253/att-02/compact.done) = No such file or directory: 0.031 sec on http probe failed: 5, giveup 2024/11/19-15:27:24.132935 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00253/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.305 (submit 269.321 ms wait 0.031 s), input 3.013 KiB, send shutdown 2024/11/19-15:27:24.141189 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00253/att-02 2024/11/19-15:27:24.141409 139851179493120 [dcompact_etcd.cc:1773] job-00253/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:24.143136 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00253) = success 2024/11/19-15:27:24.143149 139851179493120 [dcompact_etcd.cc:1773] job-00253/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:27:24.143172 139851179493120 (Original Log Time 2024/11/19-15:27:23.062957) EVENT_LOG_v1 {"time_micros": 1732001243062950, "job": 253, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5250], "score": 9.64526, "input_data_size": 7923, "oldest_snapshot_seqno": -1} 2024/11/19-15:27:24.143182 139851179493120 [compaction_job.cc:2437] [default] [JOB 253] Compacting 1@1 files to L2, score 9.65, subcompactions 1 : 1 2024/11/19-15:27:24.143193 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 251 Base level 1, inputs: [5250(7923B)] 2024/11/19-15:27:24.143463 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005410.sst) /storage/mytopling/..//tmp/Topling-82038-1Z6k9G 2024/11/19-15:27:24.143604 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005410.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.008 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.000 GB 2024/11/19-15:27:24.143757 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005410.sst): index pass time = 0.00's, 0.700'MB/sec index type = UintIndex_IL_256_32 usrkeys = 23 min-keylen = 4 max-keylen = 4 prefix = 4 subcplen = 3 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 11.83 2024/11/19-15:27:24.143761 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005410.sst): first pass time = 0.00's, 10.822'MB/sec 2024/11/19-15:27:24.143803 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005410.sst): old prealloc_size = 8715, real_size = 3440 2024/11/19-15:27:24.143823 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005410.sst): index type = UintIndex_IL_256_32 , store type = PlainBlobStore usrkeys = 23 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 4.00 avg-zkey = 11.83 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 118.13 avg-zval = 128.70 2024/11/19-15:27:24.144128 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005410.sst): second pass time = 0.00's,43126.984'MB/sec, value only(88.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 14.970'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 13.495'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.118'MB/sec write SST data time = 0.00's, 19.571'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 43126.984'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 23 avg-key = 16.00 avg-zkey = 13.91 avg-val = 118.13 avg-zval = 128.70 usrkeys = 23 avg-key = 8.00 avg-zkey = 13.91 avg-val = 125.13 avg-zval = 128.70 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 23 bytes = 80 } seq expand size = 161 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0000 GB } UnZip/Zip{ index = 0.5750 value = 0.9179 dict = 0.00 all = 0.6780 } Zip/UnZip{ index = 1.7391 value = 1.0894 dict = 0.00 all = 1.4749 } ---------------------------- total value len = 17.246374 GB avg = 0.262 KB (by entry num) total key len = 1.701658 GB avg = 0.026 KB total ukey len = 1.174288 GB avg = 0.018 KB total ukey num = 0.065921244 Billion total entry num = 0.065921244 Billion write speed all = 35.942518923 MB/sec (with seq num) write speed all = 34.942150922 MB/sec (without seq num) 2024/11/19-15:27:24.144353 139851179493120 [compaction_job.cc:1948] [default] [JOB 253] Generated table #5410: 23 keys, 4550 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:24.144403 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001244144374, "cf_name": "default", "job": 253, "event": "table_file_creation", "file_number": 5410, "file_size": 4550, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 140552265, "largest_seqno": 150747464, "table_properties": {"data_size": 2960, "index_size": 320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 368, "raw_average_key_size": 16, "raw_value_size": 2717, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 23, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 16, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-UintIndex_IL_256_32-PlainBlobStore", "creation_time": 1732001244, "oldest_key_time": 0, "file_creation_time": 1732001244, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5410, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:24.144513 139851179493120 [compaction_job.cc:703] job-00253: subcompact[0], size: 0.000005 G, files: /storage/mytopling/.rocksdb [005410] 2024/11/19-15:27:24.145414 139851179493120 (Original Log Time 2024/11/19-15:27:24.143228) EVENT_LOG_v1 {"time_micros": 1732001244143209, "job": 253, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5250], "score": 9.64526, "input_data_size": 7923, "oldest_snapshot_seqno": -1} 2024/11/19-15:27:24.145426 139851179493120 (Original Log Time 2024/11/19-15:27:24.144743) [compaction_job.cc:2022] [default] [JOB 253] Compacted 1@1 files to L2 => 4550 bytes 2024/11/19-15:27:24.145432 139851179493120 (Original Log Time 2024/11/19-15:27:24.145288) [compaction_job.cc:1224] [default] compacted to: files[0 158 75 70 0 0 0] max score 9.65 (209 files need compaction), MB/sec: 6.1 rd, 3.5 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(1.6) write-amplify(0.6) OK, records in: 23, records dropped: 0 output_compression: Snappy 2024/11/19-15:27:24.145451 139851179493120 (Original Log Time 2024/11/19-15:27:24.145314) EVENT_LOG_v1 {"time_micros": 1732001244145300, "cf": "default", "job": 253, "event": "compaction_finished", "compaction_time_micros": 1291, "compaction_time_cpu_micros": 1165, "output_level": 2, "num_output_files": 1, "total_output_size": 4550, "num_input_records": 23, "num_output_records": 23, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 158, 75, 70, 0, 0, 0]} 2024/11/19-15:27:24.145562 139851179493120 [compaction_job.cc:2437] [default] [JOB 254] Compacting 4@1 files to L2, score 9.65, subcompactions 1 : 1 2024/11/19-15:27:24.145574 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 252 Base level 1, inputs: [5256(31MB) 5258(31MB) 5261(31MB) 5264(31MB)] 2024/11/19-15:27:24.368218 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-02 2024/11/19-15:27:24.368342 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00246/att-02/compact.done) = No such file or directory: 5.001 sec on http probe failed: 1, retry 2024/11/19-15:27:24.446465 139851179493120 [dcompact_etcd.cc:1163] job-00254/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.300 sec, input = 127.234 MiB 2024/11/19-15:27:24.591311 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-00 2024/11/19-15:27:24.591403 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00250/att-00/compact.done) = No such file or directory: 13.033 sec on http probe failed: 5, giveup 2024/11/19-15:27:24.591428 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00250/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 224.039 sec, real 13.326 (submit 280.437 ms wait 13.033 s), input 224.039 MiB, send shutdown 2024/11/19-15:27:24.602969 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-00 2024/11/19-15:27:24.603227 139851158521600 [dcompact_etcd.cc:1773] job-00250/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:24.917373 139851158521600 [dcompact_etcd.cc:1163] job-00250/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.314 sec, input = 224.039 MiB 2024/11/19-15:27:25.026957 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005409.sst): old prealloc_size = 147639500, real_size = 40352480 2024/11/19-15:27:25.027056 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005409.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129429 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.00 avg-zval = 309.83 2024/11/19-15:27:25.027111 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005409.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.992 sec, Key+Value = 0.134 GB 2024/11/19-15:27:25.142681 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005409.sst): second pass time = 0.99's, 132.662'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 360.401'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2700.832'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 0.379'MB/sec write SST data time = 0.03's, 1374.581'MB/sec dict compress time = 0.00's, 1883.613'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 20449.425'MB/sec zip my value time = 0.99's, unzip length = 0.132'GB zip my value throughput = 132.662'MB/sec zip pipeline throughput = 132.662'MB/sec entries = 129429 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.00 avg-zval = 309.83 usrkeys = 129429 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.00 avg-zval = 309.83 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0980 value = 3.2824 dict = 1.00 all = 3.2187 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 17.378003 GB avg = 0.263 KB (by entry num) total key len = 1.704246 GB avg = 0.026 KB total ukey len = 1.175841 GB avg = 0.018 KB total ukey num = 0.066050673 Billion total entry num = 0.066050673 Billion write speed all = 36.128683475 MB/sec (with seq num) write speed all = 35.128246351 MB/sec (without seq num) 2024/11/19-15:27:25.203077 139851127064320 [compaction_job.cc:1948] [default] [JOB 245] Generated table #5409: 129429 keys, 41699557 bytes, temperature: kUnknown 2024/11/19-15:27:25.203134 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001245203109, "cf_name": "default", "job": 245, "event": "table_file_creation", "file_number": 5409, "file_size": 41699557, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 40101136, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588580, "raw_average_key_size": 20, "raw_value_size": 131629401, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129429, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001243, "oldest_key_time": 0, "file_creation_time": 1732001243, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5409, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:25.224159 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005411.sst) /storage/mytopling/..//tmp/Topling-82038-MJhOeB 2024/11/19-15:27:25.270758 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005411.sst): first pass time = 0.05's, 688.837'MB/sec 2024/11/19-15:27:25.270764 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005411.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.032 GB 2024/11/19-15:27:25.270796 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005411.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, dictZip workingMem = 0.0019 GB, waited 0.000 sec, Key+Value = 0.032 GB 2024/11/19-15:27:25.270872 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005411.sst): index pass time = 0.00's,3159.521'MB/sec index type = UintIndex_AllOne usrkeys = 30940 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0002 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.01 2024/11/19-15:27:25.577848 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-01 2024/11/19-15:27:25.577963 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00249/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:27:25.593885 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005411.sst): old prealloc_size = 147639500, real_size = 9709560 2024/11/19-15:27:25.593929 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005411.sst): index type = UintIndex_AllOne , store type = DictZipBlobStore usrkeys = 30940 prefix = 4 raw-key = 0.0002 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.01 raw-val = 0.0315 GB zip-val = 0.0097 GB avg-val =1017.06 avg-zval = 313.56 2024/11/19-15:27:25.605555 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005411.sst): second pass time = 0.32's, 97.404'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.01's, 859.386'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 6.235'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 1.608'MB/sec write SST data time = 0.01's, 1469.692'MB/sec dict compress time = 0.00's, 1079.063'MB/sec z-dict build time = 0.00's, sample length = 0.311'MB, throughput = 4723.844'MB/sec zip my value time = 0.32's, unzip length = 0.031'GB zip my value throughput = 97.404'MB/sec zip pipeline throughput = 97.404'MB/sec entries = 30940 avg-key = 20.00 avg-zkey = 0.01 avg-val = 1017.06 avg-zval = 313.56 usrkeys = 30940 avg-key = 12.00 avg-zkey = 0.01 avg-val = 1017.06 avg-zval = 313.56 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 3 UnZipSize{ index = 0.0004 GB value = 0.0315 GB dict = 0.31 MB all = 0.0321 GB } __ZipSize{ index = 0.0000 GB value = 0.0097 GB dict = 0.31 MB all = 0.0100 GB } UnZip/Zip{ index =1933.7500 value = 3.2436 dict = 1.00 all = 3.2042 } Zip/UnZip{ index = 0.0005 value = 0.3083 dict = 1.00 all = 0.3121 } ---------------------------- total value len = 17.409471 GB avg = 0.263 KB (by entry num) total key len = 1.704865 GB avg = 0.026 KB total ukey len = 1.176212 GB avg = 0.018 KB total ukey num = 0.066081613 Billion total entry num = 0.066081613 Billion write speed all = 36.157745904 MB/sec (with seq num) write speed all = 35.157716549 MB/sec (without seq num) 2024/11/19-15:27:25.620643 139851127064320 [compaction_job.cc:1948] [default] [JOB 245] Generated table #5411: 30940 keys, 10014111 bytes, temperature: kUnknown 2024/11/19-15:27:25.620695 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001245620670, "cf_name": "default", "job": 245, "event": "table_file_creation", "file_number": 5411, "file_size": 10014111, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 9701504, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 618800, "raw_average_key_size": 20, "raw_value_size": 31467914, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 30940, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-DictZipBlobStore", "creation_time": 1732001245, "oldest_key_time": 0, "file_creation_time": 1732001245, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5411, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:25.625894 139851127064320 [compaction_job.cc:703] job-00245: subcompact[0], size: 0.051714 G, files: /storage/mytopling/.rocksdb [005409,005411] 2024/11/19-15:27:25.627377 139851127064320 (Original Log Time 2024/11/19-15:27:23.812274) EVENT_LOG_v1 {"time_micros": 1732001243812260, "job": 245, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4631, 4617, 4787, 4619], "score": 2.74847, "input_data_size": 163722082, "oldest_snapshot_seqno": 149883333} 2024/11/19-15:27:25.627387 139851127064320 (Original Log Time 2024/11/19-15:27:25.626864) [compaction_job.cc:2022] [default] [JOB 245] Compacted 4@2 files to L3 => 51713668 bytes 2024/11/19-15:27:25.627391 139851127064320 (Original Log Time 2024/11/19-15:27:25.627249) [compaction_job.cc:1224] [default] compacted to: files[0 158 71 72 0 0 0] max score 9.40 (205 files need compaction), MB/sec: 90.3 rd, 28.5 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(156.1, 0.0 +0.0 blob) out(49.3 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 160369, records dropped: 0 output_compression: Snappy 2024/11/19-15:27:25.627405 139851127064320 (Original Log Time 2024/11/19-15:27:25.627275) EVENT_LOG_v1 {"time_micros": 1732001245627262, "cf": "default", "job": 245, "event": "compaction_finished", "compaction_time_micros": 1813632, "compaction_time_cpu_micros": 781609, "output_level": 3, "num_output_files": 2, "total_output_size": 51713668, "num_input_records": 160369, "num_output_records": 160369, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 158, 71, 72, 0, 0, 0]} 2024/11/19-15:27:25.634664 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005185.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:25.634691 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001245634688, "job": 245, "event": "table_file_deletion", "file_number": 5185} 2024/11/19-15:27:25.642001 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005186.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:25.642025 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001245642022, "job": 245, "event": "table_file_deletion", "file_number": 5186} 2024/11/19-15:27:25.649144 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005189.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:25.649166 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001245649164, "job": 245, "event": "table_file_deletion", "file_number": 5189} 2024/11/19-15:27:25.656006 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005190.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:25.656025 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001245656024, "job": 245, "event": "table_file_deletion", "file_number": 5190} 2024/11/19-15:27:25.656180 139851127064320 [compaction_job.cc:2437] [default] [JOB 255] Compacting 4@1 files to L2, score 9.40, subcompactions 1 : 1 2024/11/19-15:27:25.656190 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 253 Base level 1, inputs: [5267(31MB) 5269(31MB) 5271(31MB) 5273(31MB)] 2024/11/19-15:27:25.963118 139851127064320 [dcompact_etcd.cc:1163] job-00255/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.307 sec, input = 127.240 MiB 2024/11/19-15:27:25.983290 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-01 2024/11/19-15:27:25.983352 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00248/att-01/compact.done) = No such file or directory: 13.027 sec on http probe failed: 5, giveup 2024/11/19-15:27:25.983368 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00248/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.822 sec, real 13.339 (submit 303.895 ms wait 13.027 s), input 253.822 MiB, send shutdown 2024/11/19-15:27:25.993119 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-01 2024/11/19-15:27:25.993330 139850978166528 [dcompact_etcd.cc:1773] job-00248/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:26.248882 139850978166528 [dcompact_etcd.cc:1163] job-00248/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.255 sec, input = 253.822 MiB 2024/11/19-15:27:26.377557 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-02 2024/11/19-15:27:26.377632 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00246/att-02/compact.done) = No such file or directory: 7.015 sec on http probe failed: 2, retry 2024/11/19-15:27:27.585253 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-01 2024/11/19-15:27:27.585386 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00249/att-01/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:27:27.633523 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-00 2024/11/19-15:27:27.633614 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00254/att-00/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:27:28.386359 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-02 2024/11/19-15:27:28.386465 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00246/att-02/compact.done) = No such file or directory: 9.024 sec on http probe failed: 3, retry 2024/11/19-15:27:28.611529 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-02 2024/11/19-15:27:28.611603 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00247/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:27:29.153992 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-00 2024/11/19-15:27:29.154102 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00255/att-00/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:27:29.594915 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-01 2024/11/19-15:27:29.595029 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00249/att-01/compact.done) = No such file or directory: 9.016 sec on http probe failed: 3, retry 2024/11/19-15:27:29.641422 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-00 2024/11/19-15:27:29.641548 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00254/att-00/compact.done) = No such file or directory: 5.187 sec on http probe failed: 2, retry 2024/11/19-15:27:29.927233 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-01 2024/11/19-15:27:29.927338 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00250/att-01/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:27:30.395227 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-02 2024/11/19-15:27:30.395303 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00246/att-02/compact.done) = No such file or directory: 11.033 sec on http probe failed: 4, retry 2024/11/19-15:27:30.617616 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-02 2024/11/19-15:27:30.617711 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00247/att-02/compact.done) = No such file or directory: 7.009 sec on http probe failed: 2, retry 2024/11/19-15:27:31.162616 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-00 2024/11/19-15:27:31.162725 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00255/att-00/compact.done) = No such file or directory: 5.191 sec on http probe failed: 2, retry 2024/11/19-15:27:31.254049 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-02 2024/11/19-15:27:31.254126 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00248/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:27:31.599993 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-01 2024/11/19-15:27:31.600084 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00249/att-01/compact.done) = No such file or directory: 11.026 sec on http probe failed: 4, retry 2024/11/19-15:27:31.646446 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-00 2024/11/19-15:27:31.646510 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00254/att-00/compact.done) = No such file or directory: 7.195 sec on http probe failed: 3, retry 2024/11/19-15:27:31.932508 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-01 2024/11/19-15:27:31.932589 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00250/att-01/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:27:32.405221 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-02 2024/11/19-15:27:32.405298 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00246/att-02/compact.done) = No such file or directory: 13.042 sec on http probe failed: 5, giveup 2024/11/19-15:27:32.405317 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00246/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 222.689 sec, real 13.334 (submit 282.284 ms wait 13.042 s), input 222.689 MiB, send shutdown 2024/11/19-15:27:32.414538 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00246/att-02 2024/11/19-15:27:32.414760 139851169007360 [dcompact_etcd.cc:1773] job-00246/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:32.414987 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00246) = success 2024/11/19-15:27:32.414994 139851169007360 [dcompact_etcd.cc:1773] job-00246/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:32.415012 139851169007360 (Original Log Time 2024/11/19-15:26:52.334090) EVENT_LOG_v1 {"time_micros": 1732001212334083, "job": 246, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4630, 4792, 4793, 4796], "score": 2.64521, "input_data_size": 229900636, "oldest_snapshot_seqno": -1} 2024/11/19-15:27:32.415019 139851169007360 [compaction_job.cc:2437] [default] [JOB 246] Compacting 4@2 files to L3, score 2.65, subcompactions 1 : 1 2024/11/19-15:27:32.415027 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 244 Base level 2, inputs: [4630(31MB) 4792(62MB) 4793(62MB) 4796(62MB)] 2024/11/19-15:27:32.416597 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005412.sst) /storage/mytopling/..//tmp/Topling-82038-8AKhWY 2024/11/19-15:27:32.622883 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-02 2024/11/19-15:27:32.623004 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00247/att-02/compact.done) = No such file or directory: 9.015 sec on http probe failed: 3, retry 2024/11/19-15:27:32.650953 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005412.sst): first pass time = 0.23's, 572.750'MB/sec 2024/11/19-15:27:32.650973 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005412.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:27:32.651004 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005412.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0083 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:27:32.657910 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005412.sst): index pass time = 0.01's, 149.926'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 129427 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:27:33.176989 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-00 2024/11/19-15:27:33.177109 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00255/att-00/compact.done) = No such file or directory: 7.203 sec on http probe failed: 3, retry 2024/11/19-15:27:33.271410 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-02 2024/11/19-15:27:33.271505 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00248/att-02/compact.done) = No such file or directory: 7.007 sec on http probe failed: 2, retry 2024/11/19-15:27:33.581979 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005412.sst): old prealloc_size = 147639500, real_size = 40354336 2024/11/19-15:27:33.582089 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005412.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129427 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.02 avg-zval = 309.85 2024/11/19-15:27:33.582168 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005412.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.931 sec, Key+Value = 0.134 GB 2024/11/19-15:27:33.606108 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-01 2024/11/19-15:27:33.606212 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00249/att-01/compact.done) = No such file or directory: 13.031 sec on http probe failed: 5, giveup 2024/11/19-15:27:33.606237 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00249/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.643 sec, real 13.333 (submit 295.563 ms wait 13.031 s), input 253.643 MiB, send shutdown 2024/11/19-15:27:33.611452 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-01 2024/11/19-15:27:33.611720 139851148035840 [dcompact_etcd.cc:1773] job-00249/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:33.654359 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-00 2024/11/19-15:27:33.654473 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00254/att-00/compact.done) = No such file or directory: 9.200 sec on http probe failed: 4, retry 2024/11/19-15:27:33.699267 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005412.sst): second pass time = 0.93's, 141.391'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 355.603'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2496.731'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 0.375'MB/sec write SST data time = 0.03's, 1351.419'MB/sec dict compress time = 0.00's, 1967.123'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 15200.364'MB/sec zip my value time = 0.93's, unzip length = 0.132'GB zip my value throughput = 141.391'MB/sec zip pipeline throughput = 141.391'MB/sec entries = 129427 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.02 avg-zval = 309.85 usrkeys = 129427 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.02 avg-zval = 309.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0979 value = 3.2823 dict = 1.00 all = 3.2185 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 17.541101 GB avg = 0.265 KB (by entry num) total key len = 1.707454 GB avg = 0.026 KB total ukey len = 1.177765 GB avg = 0.018 KB total ukey num = 0.066211040 Billion total entry num = 0.066211040 Billion write speed all = 35.862565955 MB/sec (with seq num) write speed all = 34.875687560 MB/sec (without seq num) 2024/11/19-15:27:33.706734 139851169007360 [compaction_job.cc:1948] [default] [JOB 246] Generated table #5412: 129427 keys, 41701774 bytes, temperature: kUnknown 2024/11/19-15:27:33.706786 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001253706763, "cf_name": "default", "job": 246, "event": "table_file_creation", "file_number": 5412, "file_size": 41701774, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 40102992, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588540, "raw_average_key_size": 20, "raw_value_size": 131629426, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129427, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001252, "oldest_key_time": 0, "file_creation_time": 1732001252, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5412, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:33.728136 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005413.sst) /storage/mytopling/..//tmp/Topling-82038-oHqqjp 2024/11/19-15:27:33.892663 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005413.sst): first pass time = 0.16's, 603.544'MB/sec 2024/11/19-15:27:33.892668 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005413.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.099 GB 2024/11/19-15:27:33.892709 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005413.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, dictZip workingMem = 0.0061 GB, waited 0.000 sec, Key+Value = 0.099 GB 2024/11/19-15:27:33.892821 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005413.sst): index pass time = 0.00's,6535.371'MB/sec index type = UintIndex_AllOne usrkeys = 95744 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0008 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:27:33.907892 139851148035840 [dcompact_etcd.cc:1163] job-00249/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.296 sec, input = 253.643 MiB 2024/11/19-15:27:33.940795 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-01 2024/11/19-15:27:33.940850 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00250/att-01/compact.done) = No such file or directory: 9.015 sec on http probe failed: 3, retry 2024/11/19-15:27:34.597428 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005413.sst): old prealloc_size = 147639500, real_size = 29734584 2024/11/19-15:27:34.597471 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005413.sst): index type = UintIndex_AllOne , store type = DictZipBlobStore usrkeys = 95744 prefix = 4 raw-key = 0.0008 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0974 GB zip-val = 0.0297 GB avg-val =1017.01 avg-zval = 310.31 2024/11/19-15:27:34.630766 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-02 2024/11/19-15:27:34.630868 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00247/att-02/compact.done) = No such file or directory: 11.021 sec on http probe failed: 4, retry 2024/11/19-15:27:34.632190 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005413.sst): second pass time = 0.70's, 138.178'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.03's, 884.244'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 8.004'MB/sec (index lex order gen) rebuild zvType time = 0.01's, 1.683'MB/sec write SST data time = 0.02's, 1498.372'MB/sec dict compress time = 0.00's, 1364.319'MB/sec z-dict build time = 0.00's, sample length = 1.012'MB, throughput = 8928.679'MB/sec zip my value time = 0.70's, unzip length = 0.097'GB zip my value throughput = 138.178'MB/sec zip pipeline throughput = 138.178'MB/sec entries = 95744 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.01 avg-zval = 310.31 usrkeys = 95744 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.01 avg-zval = 310.31 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 3 UnZipSize{ index = 0.0011 GB value = 0.0974 GB dict = 1.01 MB all = 0.0993 GB } __ZipSize{ index = 0.0000 GB value = 0.0297 GB dict = 1.01 MB all = 0.0307 GB } UnZip/Zip{ index =5984.0000 value = 3.2774 dict = 1.00 all = 3.2316 } Zip/UnZip{ index = 0.0002 value = 0.3051 dict = 1.00 all = 0.3094 } ---------------------------- total value len = 17.638474 GB avg = 0.266 KB (by entry num) total key len = 1.709369 GB avg = 0.026 KB total ukey len = 1.178914 GB avg = 0.018 KB total ukey num = 0.066306784 Billion total entry num = 0.066306784 Billion write speed all = 35.985004559 MB/sec (with seq num) write speed all = 34.998413952 MB/sec (without seq num) 2024/11/19-15:27:34.633609 139851169007360 [compaction_job.cc:1948] [default] [JOB 246] Generated table #5413: 95744 keys, 30723636 bytes, temperature: kUnknown 2024/11/19-15:27:34.633657 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001254633634, "cf_name": "default", "job": 246, "event": "table_file_creation", "file_number": 5413, "file_size": 30723636, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 29710272, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1914880, "raw_average_key_size": 20, "raw_value_size": 97373079, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 95744, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-DictZipBlobStore", "creation_time": 1732001253, "oldest_key_time": 0, "file_creation_time": 1732001253, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5413, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:34.649321 139851169007360 [compaction_job.cc:703] job-00246: subcompact[0], size: 0.072425 G, files: /storage/mytopling/.rocksdb [005412,005413] 2024/11/19-15:27:34.651092 139851169007360 (Original Log Time 2024/11/19-15:27:32.415053) EVENT_LOG_v1 {"time_micros": 1732001252415038, "job": 246, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4630, 4792, 4793, 4796], "score": 2.64521, "input_data_size": 229900636, "oldest_snapshot_seqno": -1} 2024/11/19-15:27:34.651102 139851169007360 (Original Log Time 2024/11/19-15:27:34.650502) [compaction_job.cc:2022] [default] [JOB 246] Compacted 4@2 files to L3 => 72425410 bytes 2024/11/19-15:27:34.651106 139851169007360 (Original Log Time 2024/11/19-15:27:34.650880) [compaction_job.cc:1224] [default] compacted to: files[0 158 67 74 0 0 0] max score 9.15 (201 files need compaction), MB/sec: 102.9 rd, 32.4 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(219.3, 0.0 +0.0 blob) out(69.1 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 225171, records dropped: 0 output_compression: Snappy 2024/11/19-15:27:34.651121 139851169007360 (Original Log Time 2024/11/19-15:27:34.650906) EVENT_LOG_v1 {"time_micros": 1732001254650893, "cf": "default", "job": 246, "event": "compaction_finished", "compaction_time_micros": 2234280, "compaction_time_cpu_micros": 946538, "output_level": 3, "num_output_files": 2, "total_output_size": 72425410, "num_input_records": 225171, "num_output_records": 225171, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 158, 67, 74, 0, 0, 0]} 2024/11/19-15:27:34.657704 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005193.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:34.657729 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001254657726, "job": 246, "event": "table_file_deletion", "file_number": 5193} 2024/11/19-15:27:34.664229 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005194.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:34.664251 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001254664248, "job": 246, "event": "table_file_deletion", "file_number": 5194} 2024/11/19-15:27:34.670904 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005197.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:34.670934 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001254670932, "job": 246, "event": "table_file_deletion", "file_number": 5197} 2024/11/19-15:27:34.677562 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005199.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:34.677584 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001254677582, "job": 246, "event": "table_file_deletion", "file_number": 5199} 2024/11/19-15:27:34.677755 139851169007360 [compaction_job.cc:2437] [default] [JOB 256] Compacting 4@1 files to L2, score 9.15, subcompactions 1 : 1 2024/11/19-15:27:34.677764 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 254 Base level 1, inputs: [5276(31MB) 5279(31MB) 5282(31MB) 5284(31MB)] 2024/11/19-15:27:34.978706 139851169007360 [dcompact_etcd.cc:1163] job-00256/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.300 sec, input = 127.238 MiB 2024/11/19-15:27:35.184858 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-00 2024/11/19-15:27:35.184938 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00255/att-00/compact.done) = No such file or directory: 9.214 sec on http probe failed: 4, retry 2024/11/19-15:27:35.279239 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-02 2024/11/19-15:27:35.279319 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00248/att-02/compact.done) = No such file or directory: 9.023 sec on http probe failed: 3, retry 2024/11/19-15:27:35.663155 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-00 2024/11/19-15:27:35.663260 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00254/att-00/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:27:35.663279 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00254/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.234 sec, real 11.517 (submit 300.311 ms wait 11.208 s), input 127.234 MiB, send shutdown 2024/11/19-15:27:35.669553 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-00 2024/11/19-15:27:35.669814 139851179493120 [dcompact_etcd.cc:1773] job-00254/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:35.946111 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-01 2024/11/19-15:27:35.946183 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00250/att-01/compact.done) = No such file or directory: 11.024 sec on http probe failed: 4, retry 2024/11/19-15:27:35.974961 139851179493120 [dcompact_etcd.cc:1163] job-00254/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.305 sec, input = 127.234 MiB 2024/11/19-15:27:36.637384 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-02 2024/11/19-15:27:36.637481 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00247/att-02/compact.done) = No such file or directory: 13.028 sec on http probe failed: 5, giveup 2024/11/19-15:27:36.637499 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00247/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.893 sec, real 13.339 (submit 303.724 ms wait 13.028 s), input 253.893 MiB, send shutdown 2024/11/19-15:27:36.646384 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00247/att-02 2024/11/19-15:27:36.646613 139851137550080 [dcompact_etcd.cc:1773] job-00247/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:36.646841 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00247) = success 2024/11/19-15:27:36.646847 139851137550080 [dcompact_etcd.cc:1773] job-00247/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:36.646864 139851137550080 (Original Log Time 2024/11/19-15:26:56.564860) EVENT_LOG_v1 {"time_micros": 1732001216564849, "job": 247, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4797, 4800, 4801, 4802], "score": 2.50023, "input_data_size": 262262166, "oldest_snapshot_seqno": 150367560} 2024/11/19-15:27:36.646871 139851137550080 [compaction_job.cc:2437] [default] [JOB 247] Compacting 4@2 files to L3, score 2.50, subcompactions 1 : 1 2024/11/19-15:27:36.646879 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 245 Base level 2, inputs: [4797(62MB) 4800(63MB) 4801(61MB) 4802(63MB)] 2024/11/19-15:27:36.649808 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005414.sst) /storage/mytopling/..//tmp/Topling-82038-b0M0TA 2024/11/19-15:27:36.817031 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005414.sst): first pass time = 0.17's, 802.713'MB/sec 2024/11/19-15:27:36.817040 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005414.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:27:36.817077 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005414.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0083 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:27:36.826677 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005414.sst): index pass time = 0.01's, 107.872'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 129430 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:27:37.201712 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-00 2024/11/19-15:27:37.201811 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00255/att-00/compact.done) = No such file or directory: 11.222 sec on http probe failed: 5, giveup 2024/11/19-15:27:37.201838 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00255/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.240 sec, real 11.545 (submit 306.510 ms wait 11.222 s), input 127.240 MiB, send shutdown 2024/11/19-15:27:37.211092 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-00 2024/11/19-15:27:37.258965 139851127064320 [dcompact_etcd.cc:1773] job-00255/att-00: CleanFiles 0.048 sec : 0.000 2024/11/19-15:27:37.305990 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-02 2024/11/19-15:27:37.306114 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00248/att-02/compact.done) = No such file or directory: 11.036 sec on http probe failed: 4, retry 2024/11/19-15:27:37.567847 139851127064320 [dcompact_etcd.cc:1163] job-00255/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.309 sec, input = 127.240 MiB 2024/11/19-15:27:37.847881 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005414.sst): old prealloc_size = 147639500, real_size = 40351584 2024/11/19-15:27:37.847991 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005414.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129430 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.00 avg-zval = 309.82 2024/11/19-15:27:37.848043 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005414.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 1.031 sec, Key+Value = 0.134 GB 2024/11/19-15:27:37.951244 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-01 2024/11/19-15:27:37.951342 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00250/att-01/compact.done) = No such file or directory: 13.029 sec on http probe failed: 5, giveup 2024/11/19-15:27:37.951361 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00250/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 224.039 sec, real 13.348 (submit 314.036 ms wait 13.029 s), input 224.039 MiB, send shutdown 2024/11/19-15:27:37.960467 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-01 2024/11/19-15:27:37.963160 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005414.sst): second pass time = 1.03's, 127.698'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 361.770'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2282.805'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 0.380'MB/sec write SST data time = 0.03's, 1387.695'MB/sec dict compress time = 0.00's, 1407.323'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 14377.034'MB/sec zip my value time = 1.03's, unzip length = 0.132'GB zip my value throughput = 127.698'MB/sec zip pipeline throughput = 127.698'MB/sec entries = 129430 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.00 avg-zval = 309.82 usrkeys = 129430 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.00 avg-zval = 309.82 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0980 value = 3.2825 dict = 1.00 all = 3.2188 } Zip/UnZip{ index = 0.1409 value = 0.3046 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 17.770104 GB avg = 0.267 KB (by entry num) total key len = 1.711957 GB avg = 0.026 KB total ukey len = 1.180467 GB avg = 0.018 KB total ukey num = 0.066436214 Billion total entry num = 0.066436214 Billion write speed all = 36.011536523 MB/sec (with seq num) write speed all = 35.029106522 MB/sec (without seq num) 2024/11/19-15:27:38.032774 139851158521600 [dcompact_etcd.cc:1773] job-00250/att-01: CleanFiles 0.072 sec : 0.000 2024/11/19-15:27:38.135385 139851137550080 [compaction_job.cc:1948] [default] [JOB 247] Generated table #5414: 129430 keys, 41699006 bytes, temperature: kUnknown 2024/11/19-15:27:38.135444 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001258135420, "cf_name": "default", "job": 247, "event": "table_file_creation", "file_number": 5414, "file_size": 41699006, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 40100240, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588600, "raw_average_key_size": 20, "raw_value_size": 131630146, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129430, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001256, "oldest_key_time": 0, "file_creation_time": 1732001256, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5414, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:38.156874 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005415.sst) /storage/mytopling/..//tmp/Topling-82038-7YTGoY 2024/11/19-15:27:38.164542 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-00 2024/11/19-15:27:38.164631 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00256/att-00/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:27:38.320802 139851158521600 [dcompact_etcd.cc:1163] job-00250/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.288 sec, input = 224.039 MiB 2024/11/19-15:27:38.330741 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005415.sst): first pass time = 0.17's, 759.326'MB/sec 2024/11/19-15:27:38.330748 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005415.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.132 GB 2024/11/19-15:27:38.330768 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005415.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0081 GB, waited 0.000 sec, Key+Value = 0.132 GB 2024/11/19-15:27:38.337094 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005415.sst): index pass time = 0.01's, 160.937'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 127295 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:27:38.922259 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-02 2024/11/19-15:27:38.922365 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00249/att-02/compact.done) = No such file or directory: 5.002 sec on http probe failed: 1, retry 2024/11/19-15:27:39.168643 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-01 2024/11/19-15:27:39.168744 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00254/att-01/compact.done) = No such file or directory: 3.183 sec on http probe failed: 1, retry 2024/11/19-15:27:39.254087 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005415.sst): old prealloc_size = 147639500, real_size = 39690304 2024/11/19-15:27:39.254184 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005415.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 127295 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1295 GB zip-val = 0.0394 GB avg-val =1017.02 avg-zval = 309.85 2024/11/19-15:27:39.254245 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005415.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.923 sec, Key+Value = 0.132 GB 2024/11/19-15:27:39.314212 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-02 2024/11/19-15:27:39.314324 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00248/att-02/compact.done) = No such file or directory: 13.057 sec on http probe failed: 5, giveup 2024/11/19-15:27:39.314349 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00248/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.822 sec, real 13.321 (submit 255.475 ms wait 13.057 s), input 253.822 MiB, send shutdown 2024/11/19-15:27:39.324401 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00248/att-02 2024/11/19-15:27:39.324659 139850978166528 [dcompact_etcd.cc:1773] job-00248/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:39.324951 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00248) = success 2024/11/19-15:27:39.324962 139850978166528 [dcompact_etcd.cc:1773] job-00248/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:39.324988 139850978166528 (Original Log Time 2024/11/19-15:26:59.267736) EVENT_LOG_v1 {"time_micros": 1732001219267729, "job": 248, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4803, 4805, 4806, 4810], "score": 2.33494, "input_data_size": 262185942, "oldest_snapshot_seqno": 150553414} 2024/11/19-15:27:39.324999 139850978166528 [compaction_job.cc:2437] [default] [JOB 248] Compacting 4@2 files to L3, score 2.33, subcompactions 1 : 1 2024/11/19-15:27:39.325011 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 246 Base level 2, inputs: [4803(61MB) 4805(62MB) 4806(62MB) 4810(63MB)] 2024/11/19-15:27:39.328598 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005416.sst) /storage/mytopling/..//tmp/Topling-82038-OhuG4Q 2024/11/19-15:27:39.382065 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005415.sst): second pass time = 0.92's, 140.217'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.13's, 320.595'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2675.182'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 0.356'MB/sec write SST data time = 0.04's, 1067.490'MB/sec dict compress time = 0.00's, 1458.328'MB/sec z-dict build time = 0.00's, sample length = 1.357'MB, throughput = 17836.482'MB/sec zip my value time = 0.92's, unzip length = 0.129'GB zip my value throughput = 140.217'MB/sec zip pipeline throughput = 140.217'MB/sec entries = 127295 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.02 avg-zval = 309.85 usrkeys = 127295 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.02 avg-zval = 309.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0015 GB value = 0.1295 GB dict = 1.36 MB all = 0.1320 GB } __ZipSize{ index = 0.0002 GB value = 0.0394 GB dict = 1.36 MB all = 0.0410 GB } UnZip/Zip{ index = 7.0951 value = 3.2823 dict = 1.00 all = 3.2183 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 17.899566 GB avg = 0.269 KB (by entry num) total key len = 1.714503 GB avg = 0.026 KB total ukey len = 1.181995 GB avg = 0.018 KB total ukey num = 0.066563509 Billion total entry num = 0.066563509 Billion write speed all = 36.160706068 MB/sec (with seq num) write speed all = 35.178968515 MB/sec (without seq num) 2024/11/19-15:27:39.383984 139851137550080 [compaction_job.cc:1948] [default] [JOB 247] Generated table #5415: 127295 keys, 41017644 bytes, temperature: kUnknown 2024/11/19-15:27:39.384050 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001259384016, "cf_name": "default", "job": 247, "event": "table_file_creation", "file_number": 5415, "file_size": 41017644, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 39442992, "index_size": 215296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2545900, "raw_average_key_size": 20, "raw_value_size": 129461903, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 127295, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001258, "oldest_key_time": 0, "file_creation_time": 1732001258, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5415, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:39.413682 139851137550080 [compaction_job.cc:703] job-00247: subcompact[0], size: 0.082717 G, files: /storage/mytopling/.rocksdb [005414,005415] 2024/11/19-15:27:39.416331 139851137550080 (Original Log Time 2024/11/19-15:27:36.646904) EVENT_LOG_v1 {"time_micros": 1732001256646891, "job": 247, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4797, 4800, 4801, 4802], "score": 2.50023, "input_data_size": 262262166, "oldest_snapshot_seqno": 150367560} 2024/11/19-15:27:39.416347 139851137550080 (Original Log Time 2024/11/19-15:27:39.415553) [compaction_job.cc:2022] [default] [JOB 247] Compacted 4@2 files to L3 => 82716650 bytes 2024/11/19-15:27:39.416353 139851137550080 (Original Log Time 2024/11/19-15:27:39.416146) [compaction_job.cc:1224] [default] compacted to: files[0 158 63 76 0 0 0] max score 8.90 (197 files need compaction), MB/sec: 94.8 rd, 29.9 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(250.1, 0.0 +0.0 blob) out(78.9 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 256725, records dropped: 0 output_compression: Snappy 2024/11/19-15:27:39.416379 139851137550080 (Original Log Time 2024/11/19-15:27:39.416184) EVENT_LOG_v1 {"time_micros": 1732001259416165, "cf": "default", "job": 247, "event": "compaction_finished", "compaction_time_micros": 2766799, "compaction_time_cpu_micros": 1106660, "output_level": 3, "num_output_files": 2, "total_output_size": 82716650, "num_input_records": 256725, "num_output_records": 256725, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 158, 63, 76, 0, 0, 0]} 2024/11/19-15:27:39.427242 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005195.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:39.427278 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001259427273, "job": 247, "event": "table_file_deletion", "file_number": 5195} 2024/11/19-15:27:39.432356 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005196.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:39.432381 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001259432378, "job": 247, "event": "table_file_deletion", "file_number": 5196} 2024/11/19-15:27:39.443054 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005198.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:39.443088 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001259443084, "job": 247, "event": "table_file_deletion", "file_number": 5198} 2024/11/19-15:27:39.452935 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005200.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:39.452967 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001259452964, "job": 247, "event": "table_file_deletion", "file_number": 5200} 2024/11/19-15:27:39.453190 139851137550080 [compaction_job.cc:2437] [default] [JOB 257] Compacting 3@1 files to L2, score 8.90, subcompactions 1 : 1 2024/11/19-15:27:39.453203 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 255 Base level 1, inputs: [5288(13MB) 5289(25MB) 5251(24MB)] 2024/11/19-15:27:39.553618 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005416.sst): first pass time = 0.22's, 596.537'MB/sec 2024/11/19-15:27:39.553621 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005416.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:27:39.553654 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005416.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0083 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:27:39.560072 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005416.sst): index pass time = 0.01's, 161.428'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 129435 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:27:39.720433 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #5417. Immutable memtables: 0. 2024/11/19-15:27:39.721085 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 258] Syncing log #5246 2024/11/19-15:27:39.721179 139850967680768 (Original Log Time 2024/11/19-15:27:39.720873) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:27:39.721186 139850967680768 [flush_job.cc:879] [default] [JOB 258] Flushing memtable with next log file: 5417 2024/11/19-15:27:39.721208 139850967680768 EVENT_LOG_v1 {"time_micros": 1732001259721203, "job": 258, "event": "flush_started", "num_memtables": 1, "num_entries": 1543467, "num_deletes": 0, "total_data_size": 461952818, "memory_usage": 542375304, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:27:39.721212 139850967680768 [flush_job.cc:910] [default] [JOB 258] Level-0 flush table #5418: started 2024/11/19-15:27:39.721640 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000033.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/005418.sst in ConvertToSST, mem_size = 517.249 M, time = 419.543 us 2024/11/19-15:27:39.786330 139851137550080 [dcompact_etcd.cc:1163] job-00257/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.333 sec, input = 60.252 MiB 2024/11/19-15:27:40.178610 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-00 2024/11/19-15:27:40.178768 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00256/att-00/compact.done) = No such file or directory: 5.189 sec on http probe failed: 2, retry 2024/11/19-15:27:40.660240 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/005418.sst): fsize = 517.251 M, time(ms): open: 0.043, seek: 0.017, finish: 0.044, meta: 0.012, Flush: 0.013, sync: 938.434, close: 0.013, all: 938.576 2024/11/19-15:27:40.665978 139850967680768 [flush_job.cc:1083] [default] [JOB 258] Flush lasted 944799 microseconds, and 629797 cpu microseconds. 2024/11/19-15:27:40.725133 139850967680768 (Original Log Time 2024/11/19-15:27:40.660298) [flush_job.cc:1029] [default] [JOB 258] Level-0 flush table #5418: 542376730 bytes OK (needs compaction) 2024/11/19-15:27:40.725150 139850967680768 (Original Log Time 2024/11/19-15:27:40.665999) [memtable_list.cc:560] [default] Level-0 commit table #5418 started 2024/11/19-15:27:40.725156 139850967680768 (Original Log Time 2024/11/19-15:27:40.725005) [memtable_list.cc:759] [default] Level-0 commit table #5418: memtable #1 done 2024/11/19-15:27:40.725161 139850967680768 (Original Log Time 2024/11/19-15:27:40.725063) EVENT_LOG_v1 {"time_micros": 1732001260725054, "job": 258, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [1, 158, 63, 76, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:27:40.725166 139850967680768 (Original Log Time 2024/11/19-15:27:40.725089) [db_impl_compaction_flush.cc:310] [default] Level summary: files[1 158 63 76 0 0 0] max score 8.77 (195 files need compaction) 2024/11/19-15:27:40.758503 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-01 2024/11/19-15:27:40.758585 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00255/att-01/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:27:40.794754 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005416.sst): old prealloc_size = 147639500, real_size = 40347584 2024/11/19-15:27:40.794854 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005416.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129435 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1016.96 avg-zval = 309.78 2024/11/19-15:27:40.794905 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005416.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 1.241 sec, Key+Value = 0.134 GB 2024/11/19-15:27:40.911948 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005416.sst): second pass time = 1.24's, 106.061'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 355.863'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2618.983'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 0.380'MB/sec write SST data time = 0.03's, 1310.261'MB/sec dict compress time = 0.00's, 1164.691'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 14251.453'MB/sec zip my value time = 1.24's, unzip length = 0.132'GB zip my value throughput = 106.061'MB/sec zip pipeline throughput = 106.061'MB/sec entries = 129435 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.96 avg-zval = 309.78 usrkeys = 129435 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1016.96 avg-zval = 309.78 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0983 value = 3.2828 dict = 1.00 all = 3.2191 } Zip/UnZip{ index = 0.1409 value = 0.3046 dict = 1.00 all = 0.3106 } ---------------------------- total value len = 18.031196 GB avg = 0.270 KB (by entry num) total key len = 1.717092 GB avg = 0.026 KB total ukey len = 1.183548 GB avg = 0.018 KB total ukey num = 0.066692944 Billion total entry num = 0.066692944 Billion write speed all = 36.305751741 MB/sec (with seq num) write speed all = 35.324871761 MB/sec (without seq num) 2024/11/19-15:27:40.913108 139850978166528 [compaction_job.cc:1948] [default] [JOB 248] Generated table #5416: 129435 keys, 41694829 bytes, temperature: kUnknown 2024/11/19-15:27:40.913159 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001260913137, "cf_name": "default", "job": 248, "event": "table_file_creation", "file_number": 5416, "file_size": 41694829, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 40096240, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588700, "raw_average_key_size": 20, "raw_value_size": 131629922, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129435, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001259, "oldest_key_time": 0, "file_creation_time": 1732001259, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5416, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:40.928386 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-02 2024/11/19-15:27:40.928483 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00249/att-02/compact.done) = No such file or directory: 7.015 sec on http probe failed: 2, retry 2024/11/19-15:27:40.933570 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005419.sst) /storage/mytopling/..//tmp/Topling-82038-PFn55G 2024/11/19-15:27:41.118330 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005419.sst): first pass time = 0.18's, 714.158'MB/sec 2024/11/19-15:27:41.118329 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005419.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.132 GB 2024/11/19-15:27:41.118368 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005419.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0081 GB, waited 0.000 sec, Key+Value = 0.132 GB 2024/11/19-15:27:41.129020 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005419.sst): index pass time = 0.01's, 95.546'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 127225 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:27:41.178461 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-01 2024/11/19-15:27:41.178553 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00254/att-01/compact.done) = No such file or directory: 5.194 sec on http probe failed: 2, retry 2024/11/19-15:27:41.304992 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-00 2024/11/19-15:27:41.305128 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00257/att-00/compact.done) = No such file or directory: 1.506 sec on http probe failed: 1, retry 2024/11/19-15:27:42.049891 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005419.sst): old prealloc_size = 147639500, real_size = 39669216 2024/11/19-15:27:42.049999 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005419.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 127225 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1294 GB zip-val = 0.0394 GB avg-val =1017.00 avg-zval = 309.86 2024/11/19-15:27:42.050073 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005419.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.932 sec, Key+Value = 0.132 GB 2024/11/19-15:27:42.165141 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005419.sst): second pass time = 0.93's, 138.904'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 355.776'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2557.535'MB/sec (index lex order gen) rebuild zvType time = 0.08's, 0.378'MB/sec write SST data time = 0.03's, 1324.210'MB/sec dict compress time = 0.00's, 1445.752'MB/sec z-dict build time = 0.00's, sample length = 1.357'MB, throughput = 11510.868'MB/sec zip my value time = 0.93's, unzip length = 0.129'GB zip my value throughput = 138.904'MB/sec zip pipeline throughput = 138.904'MB/sec entries = 127225 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.00 avg-zval = 309.86 usrkeys = 127225 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.00 avg-zval = 309.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0015 GB value = 0.1294 GB dict = 1.36 MB all = 0.1319 GB } __ZipSize{ index = 0.0002 GB value = 0.0394 GB dict = 1.36 MB all = 0.0410 GB } UnZip/Zip{ index = 7.0975 value = 3.2821 dict = 1.00 all = 3.2182 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 18.160584 GB avg = 0.272 KB (by entry num) total key len = 1.719636 GB avg = 0.026 KB total ukey len = 1.185075 GB avg = 0.018 KB total ukey num = 0.066820169 Billion total entry num = 0.066820169 Billion write speed all = 36.464289382 MB/sec (with seq num) write speed all = 35.483797219 MB/sec (without seq num) 2024/11/19-15:27:42.166254 139850978166528 [compaction_job.cc:1948] [default] [JOB 248] Generated table #5419: 127225 keys, 40995742 bytes, temperature: kUnknown 2024/11/19-15:27:42.166304 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001262166282, "cf_name": "default", "job": 248, "event": "table_file_creation", "file_number": 5419, "file_size": 40995742, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 39422096, "index_size": 215104, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2544500, "raw_average_key_size": 20, "raw_value_size": 129388040, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 127225, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001260, "oldest_key_time": 0, "file_creation_time": 1732001260, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5419, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:42.184504 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-00 2024/11/19-15:27:42.184565 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00256/att-00/compact.done) = No such file or directory: 7.200 sec on http probe failed: 3, retry 2024/11/19-15:27:42.187779 139850978166528 [compaction_job.cc:703] job-00248: subcompact[0], size: 0.082691 G, files: /storage/mytopling/.rocksdb [005416,005419] 2024/11/19-15:27:42.191363 139850978166528 (Original Log Time 2024/11/19-15:27:39.325051) EVENT_LOG_v1 {"time_micros": 1732001259325028, "job": 248, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4803, 4805, 4806, 4810], "score": 2.33494, "input_data_size": 262185942, "oldest_snapshot_seqno": 150553414} 2024/11/19-15:27:42.191376 139850978166528 (Original Log Time 2024/11/19-15:27:42.189163) [compaction_job.cc:2022] [default] [JOB 248] Compacted 4@2 files to L3 => 82690571 bytes 2024/11/19-15:27:42.191380 139850978166528 (Original Log Time 2024/11/19-15:27:42.189546) [compaction_job.cc:1224] [default] compacted to: files[1 158 59 78 0 0 0] max score 8.77 (195 files need compaction), MB/sec: 91.6 rd, 28.9 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(250.0, 0.0 +0.0 blob) out(78.9 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 256660, records dropped: 0 output_compression: Snappy 2024/11/19-15:27:42.191397 139850978166528 (Original Log Time 2024/11/19-15:27:42.189573) EVENT_LOG_v1 {"time_micros": 1732001262189561, "cf": "default", "job": 248, "event": "compaction_finished", "compaction_time_micros": 2862740, "compaction_time_cpu_micros": 1133092, "output_level": 3, "num_output_files": 2, "total_output_size": 82690571, "num_input_records": 256660, "num_output_records": 256660, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 158, 59, 78, 0, 0, 0]} 2024/11/19-15:27:42.193906 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004618.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:42.193938 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001262193935, "job": 248, "event": "table_file_deletion", "file_number": 4618} 2024/11/19-15:27:42.196762 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004628.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:42.196782 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001262196780, "job": 248, "event": "table_file_deletion", "file_number": 4628} 2024/11/19-15:27:42.199260 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004778.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:42.199279 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001262199277, "job": 248, "event": "table_file_deletion", "file_number": 4778} 2024/11/19-15:27:42.204479 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004784.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:42.204498 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001262204496, "job": 248, "event": "table_file_deletion", "file_number": 4784} 2024/11/19-15:27:42.204645 139850978166528 [compaction_job.cc:2437] [default] [JOB 259] Compacting 1@1 files to L2, score 8.77, subcompactions 1 : 1 2024/11/19-15:27:42.204653 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 257 Base level 1, inputs: [5260(5065KB)] 2024/11/19-15:27:42.497460 139850978166528 [dcompact_etcd.cc:1163] job-00259/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.292 sec, input = 4.573 MiB 2024/11/19-15:27:42.519641 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-00 2024/11/19-15:27:42.519703 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00257/att-00/compact.done) = No such file or directory: 2.724 sec on http probe failed: 2, retry 2024/11/19-15:27:42.621869 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-00 2024/11/19-15:27:42.621962 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00259/att-00/compact.done) = No such file or directory: 0.114 sec on http probe failed: 1, retry 2024/11/19-15:27:42.722477 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-00 2024/11/19-15:27:42.722535 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00259/att-00/compact.done) = No such file or directory: 0.216 sec on http probe failed: 2, retry 2024/11/19-15:27:42.767955 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-01 2024/11/19-15:27:42.768019 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00255/att-01/compact.done) = No such file or directory: 5.191 sec on http probe failed: 2, retry 2024/11/19-15:27:42.822054 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-00 2024/11/19-15:27:42.822133 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00259/att-00/compact.done) = No such file or directory: 0.317 sec on http probe failed: 3, retry 2024/11/19-15:27:42.921809 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-00 2024/11/19-15:27:42.921878 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00259/att-00/compact.done) = No such file or directory: 0.416 sec on http probe failed: 4, retry 2024/11/19-15:27:42.937660 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-02 2024/11/19-15:27:42.937722 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00249/att-02/compact.done) = No such file or directory: 9.021 sec on http probe failed: 3, retry 2024/11/19-15:27:43.021610 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-00 2024/11/19-15:27:43.021697 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00259/att-00/compact.done) = No such file or directory: 0.516 sec on http probe failed: 5, giveup 2024/11/19-15:27:43.021714 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00259/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.817 (submit 292.390 ms wait 0.516 s), input 4.573 MiB, send shutdown 2024/11/19-15:27:43.028961 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-00 2024/11/19-15:27:43.029178 139850978166528 [dcompact_etcd.cc:1773] job-00259/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:43.187641 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-01 2024/11/19-15:27:43.187713 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00254/att-01/compact.done) = No such file or directory: 7.204 sec on http probe failed: 3, retry 2024/11/19-15:27:43.312148 139850978166528 [dcompact_etcd.cc:1163] job-00259/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.283 sec, input = 4.573 MiB 2024/11/19-15:27:43.330375 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-02 2024/11/19-15:27:43.330479 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00250/att-02/compact.done) = No such file or directory: 5.000 sec on http probe failed: 1, retry 2024/11/19-15:27:43.435631 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-01 2024/11/19-15:27:43.435735 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00259/att-01/compact.done) = No such file or directory: 0.114 sec on http probe failed: 1, retry 2024/11/19-15:27:43.531907 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-01 2024/11/19-15:27:43.531975 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00259/att-01/compact.done) = No such file or directory: 0.215 sec on http probe failed: 2, retry 2024/11/19-15:27:43.632082 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-01 2024/11/19-15:27:43.632166 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00259/att-01/compact.done) = No such file or directory: 0.311 sec on http probe failed: 3, retry 2024/11/19-15:27:43.732038 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-01 2024/11/19-15:27:43.732216 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00259/att-01/compact.done) = No such file or directory: 0.412 sec on http probe failed: 4, retry 2024/11/19-15:27:43.732281 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-00 2024/11/19-15:27:43.732344 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00257/att-00/compact.done) = No such file or directory: 3.939 sec on http probe failed: 3, retry 2024/11/19-15:27:43.831357 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-01 2024/11/19-15:27:43.831444 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00259/att-01/compact.done) = No such file or directory: 0.512 sec on http probe failed: 5, giveup 2024/11/19-15:27:43.831462 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00259/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.802 (submit 282.898 ms wait 0.512 s), input 4.573 MiB, send shutdown 2024/11/19-15:27:43.840605 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-01 2024/11/19-15:27:43.840816 139850978166528 [dcompact_etcd.cc:1773] job-00259/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:44.135024 139850978166528 [dcompact_etcd.cc:1163] job-00259/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.294 sec, input = 4.573 MiB 2024/11/19-15:27:44.190551 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-00 2024/11/19-15:27:44.190658 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00256/att-00/compact.done) = No such file or directory: 9.206 sec on http probe failed: 4, retry 2024/11/19-15:27:44.257975 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-02 2024/11/19-15:27:44.258066 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00259/att-02/compact.done) = No such file or directory: 0.114 sec on http probe failed: 1, retry 2024/11/19-15:27:44.358363 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-02 2024/11/19-15:27:44.358427 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00259/att-02/compact.done) = No such file or directory: 0.215 sec on http probe failed: 2, retry 2024/11/19-15:27:44.454788 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-02 2024/11/19-15:27:44.454870 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00259/att-02/compact.done) = No such file or directory: 0.315 sec on http probe failed: 3, retry 2024/11/19-15:27:44.562448 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-02 2024/11/19-15:27:44.562580 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00259/att-02/compact.done) = No such file or directory: 0.411 sec on http probe failed: 4, retry 2024/11/19-15:27:44.664516 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-02 2024/11/19-15:27:44.664598 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00259/att-02/compact.done) = No such file or directory: 0.519 sec on http probe failed: 5, giveup 2024/11/19-15:27:44.664616 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00259/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 5.000 sec, real 0.824 (submit 294.135 ms wait 0.519 s), input 4.573 MiB, send shutdown 2024/11/19-15:27:44.672771 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00259/att-02 2024/11/19-15:27:44.672981 139850978166528 [dcompact_etcd.cc:1773] job-00259/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:44.674715 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00259) = success 2024/11/19-15:27:44.674724 139850978166528 [dcompact_etcd.cc:1773] job-00259/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:27:44.674741 139850978166528 (Original Log Time 2024/11/19-15:27:42.204665) EVENT_LOG_v1 {"time_micros": 1732001262204658, "job": 259, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5260], "score": 8.77269, "input_data_size": 5186830, "oldest_snapshot_seqno": 152550336} 2024/11/19-15:27:44.674748 139850978166528 [compaction_job.cc:2437] [default] [JOB 259] Compacting 1@1 files to L2, score 8.77, subcompactions 1 : 1 2024/11/19-15:27:44.674755 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 257 Base level 1, inputs: [5260(5065KB)] 2024/11/19-15:27:44.675013 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005420.sst) /storage/mytopling/..//tmp/Topling-82038-Vm8S1R 2024/11/19-15:27:44.721480 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005420.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0018 GB, waited 0.000 sec, Key+Value = 0.005 GB 2024/11/19-15:27:44.723852 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005420.sst): first pass time = 0.05's, 98.209'MB/sec 2024/11/19-15:27:44.731790 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005420.sst): index pass time = 0.01's, 233.084'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 199800 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 2024/11/19-15:27:44.731852 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005420.sst): old prealloc_size = 5705513, real_size = 388448 2024/11/19-15:27:44.731957 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005420.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZeroLengthBlobStore usrkeys = 199800 prefix = 4 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 12.00 avg-zkey = 1.69 raw-val = 0.0000 GB zip-val = 0.0000 GB avg-val = 0.00 avg-zval = 0.00 2024/11/19-15:27:44.732008 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005420.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.011 sec, Key+Value = 0.005 GB 2024/11/19-15:27:44.748615 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005420.sst): second pass time = 0.00's, 0.000'MB/sec, value only( 0.0% of KV) wait indexing time = 0.01's, remap KeyValue time = 0.02's, 20.328'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3279.998'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 172.754'MB/sec write SST data time = 0.02's, 20.814'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.000'GB zip my value throughput = 0.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 199800 avg-key = 24.00 avg-zkey = 1.69 avg-val = 0.00 avg-zval = 0.00 usrkeys = 199800 avg-key = 16.00 avg-zkey = 1.69 avg-val = 0.00 avg-zval = 0.00 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 5 UnZipSize{ index = 0.0032 GB value = 0.0000 GB dict = 0.00 MB all = 0.0048 GB } __ZipSize{ index = 0.0003 GB value = 0.0000 GB dict = 0.00 MB all = 0.0003 GB } UnZip/Zip{ index = 9.4513 value = 0.0000 dict = 0.00 all = 14.0773 } Zip/UnZip{ index = 0.1058 value = inf dict = 0.00 all = 0.0710 } ---------------------------- total value len = 18.160584 GB avg = 0.271 KB (by entry num) total key len = 1.724431 GB avg = 0.026 KB total ukey len = 1.188272 GB avg = 0.018 KB total ukey num = 0.067019969 Billion total entry num = 0.067019969 Billion write speed all = 36.301067295 MB/sec (with seq num) write speed all = 35.322281454 MB/sec (without seq num) 2024/11/19-15:27:44.748954 139850978166528 [compaction_job.cc:1948] [default] [JOB 259] Generated table #5420: 199800 keys, 340633 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:44.748997 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001264748975, "cf_name": "default", "job": 259, "event": "table_file_creation", "file_number": 5420, "file_size": 340633, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 80, "index_size": 338240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4795200, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 199800, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZeroLengthBlobStore", "creation_time": 1732001264, "oldest_key_time": 0, "file_creation_time": 1732001264, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5420, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:44.749580 139850978166528 [compaction_job.cc:703] job-00259: subcompact[0], size: 0.000341 G, files: /storage/mytopling/.rocksdb [005420] 2024/11/19-15:27:44.750321 139850978166528 (Original Log Time 2024/11/19-15:27:44.674782) EVENT_LOG_v1 {"time_micros": 1732001264674767, "job": 259, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5260], "score": 8.77269, "input_data_size": 5186830, "oldest_snapshot_seqno": 152550336} 2024/11/19-15:27:44.750330 139850978166528 (Original Log Time 2024/11/19-15:27:44.749855) [compaction_job.cc:2022] [default] [JOB 259] Compacted 1@1 files to L2 => 340633 bytes 2024/11/19-15:27:44.750334 139850978166528 (Original Log Time 2024/11/19-15:27:44.750217) [compaction_job.cc:1224] [default] compacted to: files[1 157 60 78 0 0 0] max score 8.76 (195 files need compaction), MB/sec: 69.3 rd, 4.6 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(4.9, 0.0 +0.0 blob) out(0.3 +0.0 blob), read-write-amplify(1.1) write-amplify(0.1) OK, records in: 199800, records dropped: 0 output_compression: Snappy 2024/11/19-15:27:44.750348 139850978166528 (Original Log Time 2024/11/19-15:27:44.750236) EVENT_LOG_v1 {"time_micros": 1732001264750226, "cf": "default", "job": 259, "event": "compaction_finished", "compaction_time_micros": 74805, "compaction_time_cpu_micros": 66551, "output_level": 2, "num_output_files": 1, "total_output_size": 340633, "num_input_records": 199800, "num_output_records": 199800, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 157, 60, 78, 0, 0, 0]} 2024/11/19-15:27:44.750431 139850978166528 [compaction_job.cc:2437] [default] [JOB 260] Compacting 1@1 files to L2, score 8.76, subcompactions 1 : 1 2024/11/19-15:27:44.750439 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 258 Base level 1, inputs: [5266(28MB)] 2024/11/19-15:27:44.773064 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-01 2024/11/19-15:27:44.773128 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00255/att-01/compact.done) = No such file or directory: 7.200 sec on http probe failed: 3, retry 2024/11/19-15:27:44.942385 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-02 2024/11/19-15:27:44.942464 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00249/att-02/compact.done) = No such file or directory: 11.030 sec on http probe failed: 4, retry 2024/11/19-15:27:44.942535 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-00 2024/11/19-15:27:44.942597 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00257/att-00/compact.done) = No such file or directory: 5.151 sec on http probe failed: 4, retry 2024/11/19-15:27:45.010610 139850978166528 [dcompact_etcd.cc:1163] job-00260/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.260 sec, input = 27.819 MiB 2024/11/19-15:27:45.196265 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-01 2024/11/19-15:27:45.196342 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00254/att-01/compact.done) = No such file or directory: 9.213 sec on http probe failed: 4, retry 2024/11/19-15:27:45.339232 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-02 2024/11/19-15:27:45.339304 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00250/att-02/compact.done) = No such file or directory: 7.010 sec on http probe failed: 2, retry 2024/11/19-15:27:45.714780 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-00 2024/11/19-15:27:45.714929 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00260/att-00/compact.done) = No such file or directory: 0.696 sec on http probe failed: 1, retry 2024/11/19-15:27:46.155467 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-00 2024/11/19-15:27:46.155579 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00257/att-00/compact.done) = No such file or directory: 6.361 sec on http probe failed: 5, giveup 2024/11/19-15:27:46.155599 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00257/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 60.252 sec, real 6.702 (submit 332.648 ms wait 6.361 s), input 60.252 MiB, send shutdown 2024/11/19-15:27:46.163956 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-00 2024/11/19-15:27:46.164186 139851137550080 [dcompact_etcd.cc:1773] job-00257/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:46.201716 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-00 2024/11/19-15:27:46.201787 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00256/att-00/compact.done) = No such file or directory: 11.212 sec on http probe failed: 5, giveup 2024/11/19-15:27:46.201805 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00256/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.238 sec, real 11.524 (submit 300.458 ms wait 11.212 s), input 127.238 MiB, send shutdown 2024/11/19-15:27:46.207504 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-00 2024/11/19-15:27:46.207705 139851169007360 [dcompact_etcd.cc:1773] job-00256/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:46.280114 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-00 2024/11/19-15:27:46.280233 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00260/att-00/compact.done) = No such file or directory: 1.261 sec on http probe failed: 2, retry 2024/11/19-15:27:46.435312 139851137550080 [dcompact_etcd.cc:1163] job-00257/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.271 sec, input = 60.252 MiB 2024/11/19-15:27:46.479358 139851169007360 [dcompact_etcd.cc:1163] job-00256/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.272 sec, input = 127.238 MiB 2024/11/19-15:27:46.783404 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-01 2024/11/19-15:27:46.783480 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00255/att-01/compact.done) = No such file or directory: 9.205 sec on http probe failed: 4, retry 2024/11/19-15:27:46.845564 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-00 2024/11/19-15:27:46.845622 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00260/att-00/compact.done) = No such file or directory: 1.826 sec on http probe failed: 3, retry 2024/11/19-15:27:46.948526 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-02 2024/11/19-15:27:46.948652 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00249/att-02/compact.done) = No such file or directory: 13.035 sec on http probe failed: 5, giveup 2024/11/19-15:27:46.948679 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00249/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 253.643 sec, real 13.337 (submit 296.057 ms wait 13.035 s), input 253.643 MiB, send shutdown 2024/11/19-15:27:46.958461 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00249/att-02 2024/11/19-15:27:46.958727 139851148035840 [dcompact_etcd.cc:1773] job-00249/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:46.958994 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00249) = success 2024/11/19-15:27:46.959005 139851148035840 [dcompact_etcd.cc:1773] job-00249/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:46.959027 139851148035840 (Original Log Time 2024/11/19-15:27:06.882712) EVENT_LOG_v1 {"time_micros": 1732001226882705, "job": 249, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4811, 4814, 4815, 4980], "score": 2.16969, "input_data_size": 261892828, "oldest_snapshot_seqno": -1} 2024/11/19-15:27:46.959039 139851148035840 [compaction_job.cc:2437] [default] [JOB 249] Compacting 4@2 files to L3, score 2.17, subcompactions 1 : 1 2024/11/19-15:27:46.959051 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 247 Base level 2, inputs: [4811(61MB) 4814(63MB) 4815(61MB) 4980(62MB)] 2024/11/19-15:27:46.959426 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005421.sst) /storage/mytopling/..//tmp/Topling-82038-kg1jdI 2024/11/19-15:27:47.194556 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005421.sst): first pass time = 0.24's, 570.872'MB/sec 2024/11/19-15:27:47.194557 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005421.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:27:47.194595 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005421.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0083 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:27:47.201250 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005421.sst): index pass time = 0.01's, 155.642'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 129426 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:27:47.202286 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-01 2024/11/19-15:27:47.202365 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00254/att-01/compact.done) = No such file or directory: 11.221 sec on http probe failed: 5, giveup 2024/11/19-15:27:47.202380 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00254/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.234 sec, real 11.532 (submit 305.050 ms wait 11.221 s), input 127.234 MiB, send shutdown 2024/11/19-15:27:47.210004 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-01 2024/11/19-15:27:47.278490 139851179493120 [dcompact_etcd.cc:1773] job-00254/att-01: CleanFiles 0.068 sec : 0.000 2024/11/19-15:27:47.349344 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-02 2024/11/19-15:27:47.349457 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00250/att-02/compact.done) = No such file or directory: 9.019 sec on http probe failed: 3, retry 2024/11/19-15:27:47.424024 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-00 2024/11/19-15:27:47.424199 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00260/att-00/compact.done) = No such file or directory: 2.392 sec on http probe failed: 4, retry 2024/11/19-15:27:47.566849 139851179493120 [dcompact_etcd.cc:1163] job-00254/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.288 sec, input = 127.234 MiB 2024/11/19-15:27:47.956334 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-01 2024/11/19-15:27:47.956467 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00257/att-01/compact.done) = No such file or directory: 1.508 sec on http probe failed: 1, retry 2024/11/19-15:27:47.988106 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-00 2024/11/19-15:27:47.988210 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00260/att-00/compact.done) = No such file or directory: 2.970 sec on http probe failed: 5, giveup 2024/11/19-15:27:47.988235 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00260/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 27.819 sec, real 3.237 (submit 259.789 ms wait 2.970 s), input 27.819 MiB, send shutdown 2024/11/19-15:27:48.000994 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-00 2024/11/19-15:27:48.093536 139850978166528 [dcompact_etcd.cc:1773] job-00260/att-00: CleanFiles 0.092 sec : 0.000 2024/11/19-15:27:48.241204 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005421.sst): old prealloc_size = 147639500, real_size = 40352160 2024/11/19-15:27:48.241306 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005421.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129426 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.03 avg-zval = 309.84 2024/11/19-15:27:48.241364 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005421.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 1.047 sec, Key+Value = 0.134 GB 2024/11/19-15:27:48.382465 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005421.sst): second pass time = 1.05's, 125.771'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.14's, 295.235'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2599.259'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 0.365'MB/sec write SST data time = 0.05's, 794.886'MB/sec dict compress time = 0.00's, 1571.767'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 22296.798'MB/sec zip my value time = 1.05's, unzip length = 0.132'GB zip my value throughput = 125.771'MB/sec zip pipeline throughput = 125.771'MB/sec entries = 129426 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.03 avg-zval = 309.84 usrkeys = 129426 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.03 avg-zval = 309.84 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0978 value = 3.2825 dict = 1.00 all = 3.2187 } Zip/UnZip{ index = 0.1409 value = 0.3046 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 18.292213 GB avg = 0.272 KB (by entry num) total key len = 1.727020 GB avg = 0.026 KB total ukey len = 1.189825 GB avg = 0.018 KB total ukey num = 0.067149395 Billion total entry num = 0.067149395 Billion write speed all = 36.305248960 MB/sec (with seq num) write speed all = 35.331035617 MB/sec (without seq num) 2024/11/19-15:27:48.427610 139850978166528 [dcompact_etcd.cc:1163] job-00260/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.334 sec, input = 27.819 MiB 2024/11/19-15:27:48.483212 139851148035840 [compaction_job.cc:1948] [default] [JOB 249] Generated table #5421: 129426 keys, 41699466 bytes, temperature: kUnknown 2024/11/19-15:27:48.483302 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001268483266, "cf_name": "default", "job": 249, "event": "table_file_creation", "file_number": 5421, "file_size": 41699466, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 40100816, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588520, "raw_average_key_size": 20, "raw_value_size": 131629480, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129426, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001266, "oldest_key_time": 0, "file_creation_time": 1732001266, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5421, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:48.525228 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005422.sst) /storage/mytopling/..//tmp/Topling-82038-psTEha 2024/11/19-15:27:48.735496 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005422.sst): first pass time = 0.21's, 626.626'MB/sec 2024/11/19-15:27:48.735499 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005422.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.132 GB 2024/11/19-15:27:48.735532 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005422.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0081 GB, waited 0.000 sec, Key+Value = 0.132 GB 2024/11/19-15:27:48.741844 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005422.sst): index pass time = 0.01's, 161.105'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 127048 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:27:48.789477 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-01 2024/11/19-15:27:48.789585 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00255/att-01/compact.done) = No such file or directory: 11.216 sec on http probe failed: 5, giveup 2024/11/19-15:27:48.789604 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00255/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.240 sec, real 11.530 (submit 308.715 ms wait 11.216 s), input 127.240 MiB, send shutdown 2024/11/19-15:27:48.797630 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-01 2024/11/19-15:27:48.814132 139851127064320 [dcompact_etcd.cc:1773] job-00255/att-01: CleanFiles 0.016 sec : 0.000 2024/11/19-15:27:49.111956 139851127064320 [dcompact_etcd.cc:1163] job-00255/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.298 sec, input = 127.240 MiB 2024/11/19-15:27:49.136325 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-01 2024/11/19-15:27:49.136413 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00260/att-01/compact.done) = No such file or directory: 0.698 sec on http probe failed: 1, retry 2024/11/19-15:27:49.175726 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-01 2024/11/19-15:27:49.175821 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00257/att-01/compact.done) = No such file or directory: 2.726 sec on http probe failed: 2, retry 2024/11/19-15:27:49.364305 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-02 2024/11/19-15:27:49.364403 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00250/att-02/compact.done) = No such file or directory: 11.029 sec on http probe failed: 4, retry 2024/11/19-15:27:49.646324 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005422.sst): old prealloc_size = 147639500, real_size = 39610592 2024/11/19-15:27:49.646424 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005422.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 127048 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1292 GB zip-val = 0.0394 GB avg-val =1016.98 avg-zval = 309.83 2024/11/19-15:27:49.646480 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005422.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.911 sec, Key+Value = 0.132 GB 2024/11/19-15:27:49.670335 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-01 2024/11/19-15:27:49.670436 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00256/att-01/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:27:49.701846 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-01 2024/11/19-15:27:49.701959 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00260/att-01/compact.done) = No such file or directory: 1.265 sec on http probe failed: 2, retry 2024/11/19-15:27:49.761474 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005422.sst): second pass time = 0.91's, 141.864'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 355.508'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2568.140'MB/sec (index lex order gen) rebuild zvType time = 0.08's, 0.375'MB/sec write SST data time = 0.03's, 1348.775'MB/sec dict compress time = 0.00's, 1857.472'MB/sec z-dict build time = 0.00's, sample length = 1.353'MB, throughput = 20570.136'MB/sec zip my value time = 0.91's, unzip length = 0.129'GB zip my value throughput = 141.864'MB/sec zip pipeline throughput = 141.864'MB/sec entries = 127048 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.98 avg-zval = 309.83 usrkeys = 127048 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1016.98 avg-zval = 309.83 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0015 GB value = 0.1292 GB dict = 1.35 MB all = 0.1317 GB } __ZipSize{ index = 0.0002 GB value = 0.0394 GB dict = 1.35 MB all = 0.0409 GB } UnZip/Zip{ index = 7.0961 value = 3.2823 dict = 1.00 all = 3.2186 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 18.421418 GB avg = 0.274 KB (by entry num) total key len = 1.729561 GB avg = 0.026 KB total ukey len = 1.191349 GB avg = 0.018 KB total ukey num = 0.067276443 Billion total entry num = 0.067276443 Billion write speed all = 36.453007928 MB/sec (with seq num) write speed all = 35.479386254 MB/sec (without seq num) 2024/11/19-15:27:49.762880 139851148035840 [compaction_job.cc:1948] [default] [JOB 249] Generated table #5422: 127048 keys, 40932972 bytes, temperature: kUnknown 2024/11/19-15:27:49.762939 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001269762906, "cf_name": "default", "job": 249, "event": "table_file_creation", "file_number": 5422, "file_size": 40932972, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 39363792, "index_size": 214848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2540960, "raw_average_key_size": 20, "raw_value_size": 129204690, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 127048, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001268, "oldest_key_time": 0, "file_creation_time": 1732001268, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5422, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:49.782857 139851148035840 [compaction_job.cc:703] job-00249: subcompact[0], size: 0.082632 G, files: /storage/mytopling/.rocksdb [005421,005422] 2024/11/19-15:27:49.784837 139851148035840 (Original Log Time 2024/11/19-15:27:46.959089) EVENT_LOG_v1 {"time_micros": 1732001266959067, "job": 249, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4811, 4814, 4815, 4980], "score": 2.16969, "input_data_size": 261892828, "oldest_snapshot_seqno": -1} 2024/11/19-15:27:49.784847 139851148035840 (Original Log Time 2024/11/19-15:27:49.784350) [compaction_job.cc:2022] [default] [JOB 249] Compacted 4@2 files to L3 => 82632438 bytes 2024/11/19-15:27:49.784851 139851148035840 (Original Log Time 2024/11/19-15:27:49.784717) [compaction_job.cc:1224] [default] compacted to: files[1 157 56 80 0 0 0] max score 8.71 (194 files need compaction), MB/sec: 92.7 rd, 29.3 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(249.8, 0.0 +0.0 blob) out(78.8 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 256474, records dropped: 0 output_compression: Snappy 2024/11/19-15:27:49.784866 139851148035840 (Original Log Time 2024/11/19-15:27:49.784740) EVENT_LOG_v1 {"time_micros": 1732001269784728, "cf": "default", "job": 249, "event": "compaction_finished", "compaction_time_micros": 2823780, "compaction_time_cpu_micros": 1128824, "output_level": 3, "num_output_files": 2, "total_output_size": 82632438, "num_input_records": 256474, "num_output_records": 256474, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 157, 56, 80, 0, 0, 0]} 2024/11/19-15:27:49.790003 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004788.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:49.790028 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001269790025, "job": 249, "event": "table_file_deletion", "file_number": 4788} 2024/11/19-15:27:49.795815 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004789.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:49.795837 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001269795834, "job": 249, "event": "table_file_deletion", "file_number": 4789} 2024/11/19-15:27:49.801661 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004790.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:49.801682 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001269801680, "job": 249, "event": "table_file_deletion", "file_number": 4790} 2024/11/19-15:27:49.807012 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004791.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:49.807032 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001269807030, "job": 249, "event": "table_file_deletion", "file_number": 4791} 2024/11/19-15:27:49.807638 139851148035840 [compaction_job.cc:2437] [default] [JOB 261] Compacting 4@1 files to L2, score 8.71, subcompactions 1 : 1 2024/11/19-15:27:49.807649 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 259 Base level 1, inputs: [5274(31MB) 5280(31MB) 5286(31MB) 5293(31MB)] 2024/11/19-15:27:50.080937 139851148035840 [dcompact_etcd.cc:1163] job-00261/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.273 sec, input = 127.819 MiB 2024/11/19-15:27:50.265755 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-01 2024/11/19-15:27:50.265854 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00260/att-01/compact.done) = No such file or directory: 1.831 sec on http probe failed: 3, retry 2024/11/19-15:27:50.385800 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-01 2024/11/19-15:27:50.385894 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00257/att-01/compact.done) = No such file or directory: 3.946 sec on http probe failed: 3, retry 2024/11/19-15:27:50.755967 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-02 2024/11/19-15:27:50.756071 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00254/att-02/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:27:50.831444 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-01 2024/11/19-15:27:50.831526 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00260/att-01/compact.done) = No such file or directory: 2.395 sec on http probe failed: 4, retry 2024/11/19-15:27:51.373724 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-02 2024/11/19-15:27:51.373817 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00250/att-02/compact.done) = No such file or directory: 13.044 sec on http probe failed: 5, giveup 2024/11/19-15:27:51.373835 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00250/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 224.039 sec, real 13.341 (submit 287.905 ms wait 13.044 s), input 224.039 MiB, send shutdown 2024/11/19-15:27:51.378904 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00250/att-02 2024/11/19-15:27:51.379128 139851158521600 [dcompact_etcd.cc:1773] job-00250/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:51.379363 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00250) = success 2024/11/19-15:27:51.379369 139851158521600 [dcompact_etcd.cc:1773] job-00250/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:51.379385 139851158521600 (Original Log Time 2024/11/19-15:27:11.265265) EVENT_LOG_v1 {"time_micros": 1732001231265258, "job": 250, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4981, 4982, 4983, 4986], "score": 2.00456, "input_data_size": 231300316, "oldest_snapshot_seqno": -1} 2024/11/19-15:27:51.379392 139851158521600 [compaction_job.cc:2437] [default] [JOB 250] Compacting 4@2 files to L3, score 2.00, subcompactions 1 : 1 2024/11/19-15:27:51.379400 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 248 Base level 2, inputs: [4981(32MB) 4982(62MB) 4983(62MB) 4986(62MB)] 2024/11/19-15:27:51.381143 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005423.sst) /storage/mytopling/..//tmp/Topling-82038-2SUHQi 2024/11/19-15:27:51.396811 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-01 2024/11/19-15:27:51.396870 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00260/att-01/compact.done) = No such file or directory: 2.960 sec on http probe failed: 5, giveup 2024/11/19-15:27:51.396885 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00260/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 27.819 sec, real 3.303 (submit 333.865 ms wait 2.960 s), input 27.819 MiB, send shutdown 2024/11/19-15:27:51.401390 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-01 2024/11/19-15:27:51.401586 139850978166528 [dcompact_etcd.cc:1773] job-00260/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:51.538323 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005423.sst): first pass time = 0.16's, 854.003'MB/sec 2024/11/19-15:27:51.538328 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005423.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:27:51.538362 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005423.sst: sumWaitingMem = 0.008 GB, sumWorkingMem = 0.001 GB, dictZip workingMem = 0.0083 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/19-15:27:51.548788 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005423.sst): index pass time = 0.01's, 99.325'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 129430 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:27:51.598980 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-01 2024/11/19-15:27:51.599063 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00257/att-01/compact.done) = No such file or directory: 5.156 sec on http probe failed: 4, retry 2024/11/19-15:27:51.678426 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-01 2024/11/19-15:27:51.678503 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00256/att-01/compact.done) = No such file or directory: 5.191 sec on http probe failed: 2, retry 2024/11/19-15:27:51.682792 139850978166528 [dcompact_etcd.cc:1163] job-00260/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.281 sec, input = 27.819 MiB 2024/11/19-15:27:52.313126 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-02 2024/11/19-15:27:52.313252 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00255/att-02/compact.done) = No such file or directory: 3.185 sec on http probe failed: 1, retry 2024/11/19-15:27:52.386083 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-02 2024/11/19-15:27:52.386203 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00260/att-02/compact.done) = No such file or directory: 0.696 sec on http probe failed: 1, retry 2024/11/19-15:27:52.504860 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005423.sst): old prealloc_size = 147639500, real_size = 40351024 2024/11/19-15:27:52.504976 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005423.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129430 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.00 avg-zval = 309.82 2024/11/19-15:27:52.505049 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005423.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.967 sec, Key+Value = 0.134 GB 2024/11/19-15:27:52.621535 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005423.sst): second pass time = 0.97's, 136.195'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 357.446'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2228.496'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 0.378'MB/sec write SST data time = 0.03's, 1343.999'MB/sec dict compress time = 0.00's, 1927.942'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 17600.419'MB/sec zip my value time = 0.97's, unzip length = 0.132'GB zip my value throughput = 136.195'MB/sec zip pipeline throughput = 136.195'MB/sec entries = 129430 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.00 avg-zval = 309.82 usrkeys = 129430 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.00 avg-zval = 309.82 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0980 value = 3.2826 dict = 1.00 all = 3.2188 } Zip/UnZip{ index = 0.1409 value = 0.3046 dict = 1.00 all = 0.3107 } ---------------------------- total value len = 18.553048 GB avg = 0.275 KB (by entry num) total key len = 1.732150 GB avg = 0.026 KB total ukey len = 1.192903 GB avg = 0.018 KB total ukey num = 0.067405873 Billion total entry num = 0.067405873 Billion write speed all = 36.506927484 MB/sec (with seq num) write speed all = 35.536453770 MB/sec (without seq num) 2024/11/19-15:27:52.622889 139851158521600 [compaction_job.cc:1948] [default] [JOB 250] Generated table #5423: 129430 keys, 41698693 bytes, temperature: kUnknown 2024/11/19-15:27:52.622949 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001272622917, "cf_name": "default", "job": 250, "event": "table_file_creation", "file_number": 5423, "file_size": 41698693, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 40099680, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588600, "raw_average_key_size": 20, "raw_value_size": 131629709, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129430, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001271, "oldest_key_time": 0, "file_creation_time": 1732001271, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5423, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:52.644348 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005424.sst) /storage/mytopling/..//tmp/Topling-82038-z3pTux 2024/11/19-15:27:52.732121 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005424.sst): first pass time = 0.09's,1147.546'MB/sec 2024/11/19-15:27:52.732132 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005424.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0005 GB, waited 0.000 sec, Key+Value = 0.101 GB 2024/11/19-15:27:52.732171 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005424.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, dictZip workingMem = 0.0061 GB, waited 0.000 sec, Key+Value = 0.101 GB 2024/11/19-15:27:52.739822 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005424.sst): index pass time = 0.01's, 101.556'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 97109 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0008 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 2024/11/19-15:27:52.761029 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-02 2024/11/19-15:27:52.761113 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00254/att-02/compact.done) = No such file or directory: 5.189 sec on http probe failed: 2, retry 2024/11/19-15:27:52.808891 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-01 2024/11/19-15:27:52.808972 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00257/att-01/compact.done) = No such file or directory: 6.369 sec on http probe failed: 5, giveup 2024/11/19-15:27:52.808990 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00257/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 60.252 sec, real 6.645 (submit 271.051 ms wait 6.369 s), input 60.252 MiB, send shutdown 2024/11/19-15:27:52.817343 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-01 2024/11/19-15:27:52.817563 139851137550080 [dcompact_etcd.cc:1773] job-00257/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:52.961660 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-02 2024/11/19-15:27:52.961755 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00260/att-02/compact.done) = No such file or directory: 1.265 sec on http probe failed: 2, retry 2024/11/19-15:27:53.093396 139851137550080 [dcompact_etcd.cc:1163] job-00257/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.276 sec, input = 60.252 MiB 2024/11/19-15:27:53.291436 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-00 2024/11/19-15:27:53.291557 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00261/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:27:53.472770 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005424.sst): old prealloc_size = 147639500, real_size = 30325040 2024/11/19-15:27:53.472857 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005424.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 97109 prefix = 4 raw-key = 0.0008 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.0988 GB zip-val = 0.0301 GB avg-val =1017.02 avg-zval = 310.33 2024/11/19-15:27:53.472935 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005424.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.741 sec, Key+Value = 0.101 GB 2024/11/19-15:27:53.523603 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-02 2024/11/19-15:27:53.523693 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00260/att-02/compact.done) = No such file or directory: 1.835 sec on http probe failed: 3, retry 2024/11/19-15:27:53.560596 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005424.sst): second pass time = 0.74's, 133.356'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 356.711'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2373.858'MB/sec (index lex order gen) rebuild zvType time = 0.06's, 0.380'MB/sec write SST data time = 0.02's, 1314.311'MB/sec dict compress time = 0.00's, 1433.980'MB/sec z-dict build time = 0.00's, sample length = 1.020'MB, throughput = 13455.979'MB/sec zip my value time = 0.74's, unzip length = 0.099'GB zip my value throughput = 133.356'MB/sec zip pipeline throughput = 133.356'MB/sec entries = 97109 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.02 avg-zval = 310.33 usrkeys = 97109 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.02 avg-zval = 310.33 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0012 GB value = 0.0988 GB dict = 1.02 MB all = 0.1007 GB } __ZipSize{ index = 0.0002 GB value = 0.0301 GB dict = 1.02 MB all = 0.0313 GB } UnZip/Zip{ index = 7.0820 value = 3.2772 dict = 1.00 all = 3.2150 } Zip/UnZip{ index = 0.1412 value = 0.3051 dict = 1.00 all = 0.3110 } ---------------------------- total value len = 18.651809 GB avg = 0.276 KB (by entry num) total key len = 1.734092 GB avg = 0.026 KB total ukey len = 1.194068 GB avg = 0.018 KB total ukey num = 0.067502982 Billion total entry num = 0.067502982 Billion write speed all = 36.626263644 MB/sec (with seq num) write speed all = 35.656031512 MB/sec (without seq num) 2024/11/19-15:27:53.561508 139851158521600 [compaction_job.cc:1948] [default] [JOB 250] Generated table #5424: 97109 keys, 31322660 bytes, temperature: kUnknown 2024/11/19-15:27:53.561557 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273561534, "cf_name": "default", "job": 250, "event": "table_file_creation", "file_number": 5424, "file_size": 31322660, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 30136032, "index_size": 164544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1942180, "raw_average_key_size": 20, "raw_value_size": 98761666, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 97109, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732001272, "oldest_key_time": 0, "file_creation_time": 1732001272, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5424, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:53.577757 139851158521600 [compaction_job.cc:703] job-00250: subcompact[0], size: 0.073021 G, files: /storage/mytopling/.rocksdb [005423,005424] 2024/11/19-15:27:53.579488 139851158521600 (Original Log Time 2024/11/19-15:27:51.379424) EVENT_LOG_v1 {"time_micros": 1732001271379411, "job": 250, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [4981, 4982, 4983, 4986], "score": 2.00456, "input_data_size": 231300316, "oldest_snapshot_seqno": -1} 2024/11/19-15:27:53.579497 139851158521600 (Original Log Time 2024/11/19-15:27:53.578971) [compaction_job.cc:2022] [default] [JOB 250] Compacted 4@2 files to L3 => 73021353 bytes 2024/11/19-15:27:53.579501 139851158521600 (Original Log Time 2024/11/19-15:27:53.579333) [compaction_job.cc:1224] [default] compacted to: files[1 157 52 82 0 0 0] max score 8.46 (190 files need compaction), MB/sec: 105.2 rd, 33.2 wr, level 3, files in(4, 0) out(2 +0 blob) MB in(220.6, 0.0 +0.0 blob) out(69.6 +0.0 blob), read-write-amplify(1.3) write-amplify(0.3) OK, records in: 226539, records dropped: 0 output_compression: Snappy 2024/11/19-15:27:53.579506 139851158521600 (Original Log Time 2024/11/19-15:27:53.579356) EVENT_LOG_v1 {"time_micros": 1732001273579344, "cf": "default", "job": 250, "event": "compaction_finished", "compaction_time_micros": 2198345, "compaction_time_cpu_micros": 979325, "output_level": 3, "num_output_files": 2, "total_output_size": 73021353, "num_input_records": 226539, "num_output_records": 226539, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 157, 52, 82, 0, 0, 0]} 2024/11/19-15:27:53.652050 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005059.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.652087 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273652083, "job": 250, "event": "table_file_deletion", "file_number": 5059} 2024/11/19-15:27:53.653023 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005097.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.653039 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273653037, "job": 250, "event": "table_file_deletion", "file_number": 5097} 2024/11/19-15:27:53.653900 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005099.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.653914 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273653913, "job": 250, "event": "table_file_deletion", "file_number": 5099} 2024/11/19-15:27:53.654757 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005101.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.654769 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273654768, "job": 250, "event": "table_file_deletion", "file_number": 5101} 2024/11/19-15:27:53.655729 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005103.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.655744 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273655742, "job": 250, "event": "table_file_deletion", "file_number": 5103} 2024/11/19-15:27:53.656601 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005105.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.656614 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273656613, "job": 250, "event": "table_file_deletion", "file_number": 5105} 2024/11/19-15:27:53.657450 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005107.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.657463 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273657461, "job": 250, "event": "table_file_deletion", "file_number": 5107} 2024/11/19-15:27:53.658309 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005109.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.658321 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273658320, "job": 250, "event": "table_file_deletion", "file_number": 5109} 2024/11/19-15:27:53.659146 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005110.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.659158 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273659157, "job": 250, "event": "table_file_deletion", "file_number": 5110} 2024/11/19-15:27:53.660000 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005111.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.660014 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273660012, "job": 250, "event": "table_file_deletion", "file_number": 5111} 2024/11/19-15:27:53.660867 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005112.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.660880 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273660878, "job": 250, "event": "table_file_deletion", "file_number": 5112} 2024/11/19-15:27:53.661715 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005114.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.661728 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273661726, "job": 250, "event": "table_file_deletion", "file_number": 5114} 2024/11/19-15:27:53.662563 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005115.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.662576 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273662574, "job": 250, "event": "table_file_deletion", "file_number": 5115} 2024/11/19-15:27:53.663484 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005116.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.663498 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273663496, "job": 250, "event": "table_file_deletion", "file_number": 5116} 2024/11/19-15:27:53.664781 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005117.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.664800 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273664798, "job": 250, "event": "table_file_deletion", "file_number": 5117} 2024/11/19-15:27:53.665725 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005119.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.665738 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273665737, "job": 250, "event": "table_file_deletion", "file_number": 5119} 2024/11/19-15:27:53.666642 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005121.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.666655 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273666654, "job": 250, "event": "table_file_deletion", "file_number": 5121} 2024/11/19-15:27:53.667545 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005122.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.667558 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273667557, "job": 250, "event": "table_file_deletion", "file_number": 5122} 2024/11/19-15:27:53.668448 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005124.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.668461 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273668459, "job": 250, "event": "table_file_deletion", "file_number": 5124} 2024/11/19-15:27:53.669497 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005125.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.669513 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273669511, "job": 250, "event": "table_file_deletion", "file_number": 5125} 2024/11/19-15:27:53.670377 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005126.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.670391 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273670389, "job": 250, "event": "table_file_deletion", "file_number": 5126} 2024/11/19-15:27:53.671249 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005127.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.671261 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273671260, "job": 250, "event": "table_file_deletion", "file_number": 5127} 2024/11/19-15:27:53.672133 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005129.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.672145 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273672144, "job": 250, "event": "table_file_deletion", "file_number": 5129} 2024/11/19-15:27:53.673657 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005130.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.673676 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273673673, "job": 250, "event": "table_file_deletion", "file_number": 5130} 2024/11/19-15:27:53.674555 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005131.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.674568 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273674567, "job": 250, "event": "table_file_deletion", "file_number": 5131} 2024/11/19-15:27:53.675450 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005132.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.675464 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273675462, "job": 250, "event": "table_file_deletion", "file_number": 5132} 2024/11/19-15:27:53.676294 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005134.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.676306 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273676305, "job": 250, "event": "table_file_deletion", "file_number": 5134} 2024/11/19-15:27:53.677161 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005135.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.677173 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273677172, "job": 250, "event": "table_file_deletion", "file_number": 5135} 2024/11/19-15:27:53.678024 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005136.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.678038 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273678036, "job": 250, "event": "table_file_deletion", "file_number": 5136} 2024/11/19-15:27:53.679049 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005138.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.679063 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273679061, "job": 250, "event": "table_file_deletion", "file_number": 5138} 2024/11/19-15:27:53.679957 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005139.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.679969 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273679968, "job": 250, "event": "table_file_deletion", "file_number": 5139} 2024/11/19-15:27:53.680843 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005140.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.680855 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273680854, "job": 250, "event": "table_file_deletion", "file_number": 5140} 2024/11/19-15:27:53.681695 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005141.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.681707 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273681705, "job": 250, "event": "table_file_deletion", "file_number": 5141} 2024/11/19-15:27:53.682564 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005143.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.682576 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273682575, "job": 250, "event": "table_file_deletion", "file_number": 5143} 2024/11/19-15:27:53.683832 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005144.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.683849 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273683848, "job": 250, "event": "table_file_deletion", "file_number": 5144} 2024/11/19-15:27:53.684697 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005145.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.684710 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273684709, "job": 250, "event": "table_file_deletion", "file_number": 5145} 2024/11/19-15:27:53.685571 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005147.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.685583 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273685582, "job": 250, "event": "table_file_deletion", "file_number": 5147} 2024/11/19-15:27:53.686440 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005149.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.686452 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273686451, "job": 250, "event": "table_file_deletion", "file_number": 5149} 2024/11/19-15:27:53.687374 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-01 2024/11/19-15:27:53.687441 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00256/att-01/compact.done) = No such file or directory: 7.199 sec on http probe failed: 3, retry 2024/11/19-15:27:53.687538 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005151.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.687554 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273687553, "job": 250, "event": "table_file_deletion", "file_number": 5151} 2024/11/19-15:27:53.688391 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005152.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.688403 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273688402, "job": 250, "event": "table_file_deletion", "file_number": 5152} 2024/11/19-15:27:53.689242 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005153.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.689254 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273689253, "job": 250, "event": "table_file_deletion", "file_number": 5153} 2024/11/19-15:27:53.690128 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005154.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.690140 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273690138, "job": 250, "event": "table_file_deletion", "file_number": 5154} 2024/11/19-15:27:53.691107 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005155.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.691122 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273691120, "job": 250, "event": "table_file_deletion", "file_number": 5155} 2024/11/19-15:27:53.691982 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005156.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.691995 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273691994, "job": 250, "event": "table_file_deletion", "file_number": 5156} 2024/11/19-15:27:53.692814 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005157.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.692825 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273692824, "job": 250, "event": "table_file_deletion", "file_number": 5157} 2024/11/19-15:27:53.693283 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005158.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.693294 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273693293, "job": 250, "event": "table_file_deletion", "file_number": 5158} 2024/11/19-15:27:53.766137 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005168.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.766170 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273766166, "job": 250, "event": "table_file_deletion", "file_number": 5168} 2024/11/19-15:27:53.834326 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005173.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.834372 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273834367, "job": 250, "event": "table_file_deletion", "file_number": 5173} 2024/11/19-15:27:53.913773 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005188.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.913815 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273913811, "job": 250, "event": "table_file_deletion", "file_number": 5188} 2024/11/19-15:27:53.981570 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005192.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:53.981610 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001273981607, "job": 250, "event": "table_file_deletion", "file_number": 5192} 2024/11/19-15:27:54.051182 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005219.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:54.051227 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001274051218, "job": 250, "event": "table_file_deletion", "file_number": 5219} 2024/11/19-15:27:54.089431 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-02 2024/11/19-15:27:54.089523 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00260/att-02/compact.done) = No such file or directory: 2.397 sec on http probe failed: 4, retry 2024/11/19-15:27:54.121339 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005232.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:54.121375 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001274121371, "job": 250, "event": "table_file_deletion", "file_number": 5232} 2024/11/19-15:27:54.202114 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005247.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:54.202163 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001274202159, "job": 250, "event": "table_file_deletion", "file_number": 5247} 2024/11/19-15:27:54.202345 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005250.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:54.202358 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001274202356, "job": 250, "event": "table_file_deletion", "file_number": 5250} 2024/11/19-15:27:54.202554 139851158521600 [compaction_job.cc:2437] [default] [JOB 262] Compacting 4@1 files to L2, score 8.46, subcompactions 1 : 1 2024/11/19-15:27:54.202564 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 260 Base level 1, inputs: [5297(31MB) 5300(31MB) 5303(31MB) 5305(31MB)] 2024/11/19-15:27:54.319430 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-02 2024/11/19-15:27:54.319557 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00255/att-02/compact.done) = No such file or directory: 5.201 sec on http probe failed: 2, retry 2024/11/19-15:27:54.501872 139851158521600 [dcompact_etcd.cc:1163] job-00262/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.299 sec, input = 127.819 MiB 2024/11/19-15:27:54.604698 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-02 2024/11/19-15:27:54.604783 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00257/att-02/compact.done) = No such file or directory: 1.506 sec on http probe failed: 1, retry 2024/11/19-15:27:54.650971 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-02 2024/11/19-15:27:54.651062 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00260/att-02/compact.done) = No such file or directory: 2.963 sec on http probe failed: 5, giveup 2024/11/19-15:27:54.651080 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00260/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 27.819 sec, real 3.249 (submit 281.137 ms wait 2.963 s), input 27.819 MiB, send shutdown 2024/11/19-15:27:54.655979 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00260/att-02 2024/11/19-15:27:54.656220 139850978166528 [dcompact_etcd.cc:1773] job-00260/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:54.656520 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00260) = success 2024/11/19-15:27:54.656525 139850978166528 [dcompact_etcd.cc:1773] job-00260/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:54.656542 139850978166528 (Original Log Time 2024/11/19-15:27:44.750450) EVENT_LOG_v1 {"time_micros": 1732001264750444, "job": 260, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5266], "score": 8.76303, "input_data_size": 29463747, "oldest_snapshot_seqno": -1} 2024/11/19-15:27:54.656555 139850978166528 [compaction_job.cc:2437] [default] [JOB 260] Compacting 1@1 files to L2, score 8.76, subcompactions 1 : 1 2024/11/19-15:27:54.656563 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 258 Base level 1, inputs: [5266(28MB)] 2024/11/19-15:27:54.656818 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005425.sst) /storage/mytopling/..//tmp/Topling-82038-ekbVKq 2024/11/19-15:27:54.769789 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-02 2024/11/19-15:27:54.769864 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00254/att-02/compact.done) = No such file or directory: 7.194 sec on http probe failed: 3, retry 2024/11/19-15:27:54.819674 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005425.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.029 GB 2024/11/19-15:27:54.854553 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005425.sst): index pass time = 0.03's, 229.307'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 666000 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0080 GB zip-key = 0.0011 GB avg-key = 12.00 avg-zkey = 1.68 2024/11/19-15:27:54.864179 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005425.sst): first pass time = 0.21's, 140.684'MB/sec 2024/11/19-15:27:54.864235 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005425.sst): old prealloc_size = 32410121, real_size = 15127952 2024/11/19-15:27:54.864527 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005425.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 666000 prefix = 4 raw-key = 0.0080 GB zip-key = 0.0011 GB avg-key = 12.00 avg-zkey = 1.68 raw-val = 0.0132 GB zip-val = 0.0138 GB avg-val = 19.80 avg-zval = 20.78 2024/11/19-15:27:54.864606 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005425.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0017 GB, waited 0.045 sec, Key+Value = 0.029 GB 2024/11/19-15:27:54.943831 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005425.sst): second pass time = 0.00's,42814285.714'MB/sec, value only(45.2% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.08's, 188.082'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3877.043'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 7.133'MB/sec write SST data time = 0.06's, 267.516'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.013'GB zip my value throughput = 42814285.714'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 666000 avg-key = 24.00 avg-zkey = 1.68 avg-val = 19.80 avg-zval = 20.78 usrkeys = 666000 avg-key = 16.00 avg-zkey = 1.68 avg-val = 19.80 avg-zval = 20.78 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 1616 UnZipSize{ index = 0.0107 GB value = 0.0132 GB dict = 0.00 MB all = 0.0292 GB } __ZipSize{ index = 0.0011 GB value = 0.0138 GB dict = 0.00 MB all = 0.0150 GB } UnZip/Zip{ index = 9.5089 value = 0.9528 dict = 0.00 all = 1.9489 } Zip/UnZip{ index = 0.1052 value = 1.0496 dict = 0.00 all = 0.5131 } ---------------------------- total value len = 18.664996 GB avg = 0.274 KB (by entry num) total key len = 1.750076 GB avg = 0.026 KB total ukey len = 1.204724 GB avg = 0.018 KB total ukey num = 0.068168982 Billion total entry num = 0.068168982 Billion write speed all = 36.587745894 MB/sec (with seq num) write speed all = 35.610370189 MB/sec (without seq num) 2024/11/19-15:27:54.945233 139850978166528 [compaction_job.cc:1948] [default] [JOB 260] Generated table #5425: 666000 keys, 14967888 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:54.945282 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001274945259, "cf_name": "default", "job": 260, "event": "table_file_creation", "file_number": 5425, "file_size": 14967888, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 13840640, "index_size": 1120640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 15984000, "raw_average_key_size": 24, "raw_value_size": 13186800, "raw_average_value_size": 19, "num_data_blocks": 1, "num_entries": 666000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001274, "oldest_key_time": 0, "file_creation_time": 1732001274, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5425, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:54.950979 139850978166528 [compaction_job.cc:703] job-00260: subcompact[0], size: 0.014968 G, files: /storage/mytopling/.rocksdb [005425] 2024/11/19-15:27:54.952015 139850978166528 (Original Log Time 2024/11/19-15:27:54.656591) EVENT_LOG_v1 {"time_micros": 1732001274656576, "job": 260, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5266], "score": 8.76303, "input_data_size": 29463747, "oldest_snapshot_seqno": -1} 2024/11/19-15:27:54.952025 139850978166528 (Original Log Time 2024/11/19-15:27:54.951531) [compaction_job.cc:2022] [default] [JOB 260] Compacted 1@1 files to L2 => 14967888 bytes 2024/11/19-15:27:54.952029 139850978166528 (Original Log Time 2024/11/19-15:27:54.951889) [compaction_job.cc:1224] [default] compacted to: files[1 156 53 82 0 0 0] max score 8.21 (187 files need compaction), MB/sec: 100.1 rd, 50.8 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(28.1, 0.0 +0.0 blob) out(14.3 +0.0 blob), read-write-amplify(1.5) write-amplify(0.5) OK, records in: 666000, records dropped: 0 output_compression: Snappy 2024/11/19-15:27:54.952046 139850978166528 (Original Log Time 2024/11/19-15:27:54.951911) EVENT_LOG_v1 {"time_micros": 1732001274951900, "cf": "default", "job": 260, "event": "compaction_finished", "compaction_time_micros": 294399, "compaction_time_cpu_micros": 293500, "output_level": 2, "num_output_files": 1, "total_output_size": 14967888, "num_input_records": 666000, "num_output_records": 666000, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 156, 53, 82, 0, 0, 0]} 2024/11/19-15:27:54.952128 139850978166528 [compaction_job.cc:2437] [default] [JOB 263] Compacting 4@1 files to L2, score 8.21, subcompactions 1 : 1 2024/11/19-15:27:54.952136 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 261 Base level 1, inputs: [5307(31MB) 5312(31MB) 5316(31MB) 5318(31MB)] 2024/11/19-15:27:55.225498 139850978166528 [dcompact_etcd.cc:1163] job-00263/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.273 sec, input = 127.818 MiB 2024/11/19-15:27:55.301173 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-00 2024/11/19-15:27:55.301265 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00261/att-00/compact.done) = No such file or directory: 5.211 sec on http probe failed: 2, retry 2024/11/19-15:27:55.693302 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-01 2024/11/19-15:27:55.693402 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00256/att-01/compact.done) = No such file or directory: 9.208 sec on http probe failed: 4, retry 2024/11/19-15:27:55.819570 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-02 2024/11/19-15:27:55.819635 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00257/att-02/compact.done) = No such file or directory: 2.716 sec on http probe failed: 2, retry 2024/11/19-15:27:56.328643 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-02 2024/11/19-15:27:56.328732 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00255/att-02/compact.done) = No such file or directory: 7.208 sec on http probe failed: 3, retry 2024/11/19-15:27:56.774773 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-02 2024/11/19-15:27:56.774849 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00254/att-02/compact.done) = No such file or directory: 9.203 sec on http probe failed: 4, retry 2024/11/19-15:27:57.033410 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-02 2024/11/19-15:27:57.033489 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00257/att-02/compact.done) = No such file or directory: 3.931 sec on http probe failed: 3, retry 2024/11/19-15:27:57.306783 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-00 2024/11/19-15:27:57.306880 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00261/att-00/compact.done) = No such file or directory: 7.220 sec on http probe failed: 3, retry 2024/11/19-15:27:57.702818 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-01 2024/11/19-15:27:57.702911 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00256/att-01/compact.done) = No such file or directory: 11.214 sec on http probe failed: 5, giveup 2024/11/19-15:27:57.702938 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00256/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.238 sec, real 11.495 (submit 271.581 ms wait 11.214 s), input 127.238 MiB, send shutdown 2024/11/19-15:27:57.706844 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-00 2024/11/19-15:27:57.706917 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00262/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:27:57.712467 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-01 2024/11/19-15:27:57.712688 139851169007360 [dcompact_etcd.cc:1773] job-00256/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:58.018464 139851169007360 [dcompact_etcd.cc:1163] job-00256/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.306 sec, input = 127.238 MiB 2024/11/19-15:27:58.019842 139852349699840 [db_impl_write.cc:2181] [default] New memtable created with log file: #5426. Immutable memtables: 0. 2024/11/19-15:27:58.020495 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 264] Syncing log #5417 2024/11/19-15:27:58.020590 139850957195008 (Original Log Time 2024/11/19-15:27:58.020292) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:27:58.020604 139850957195008 [flush_job.cc:879] [default] [JOB 264] Flushing memtable with next log file: 5426 2024/11/19-15:27:58.020631 139850957195008 EVENT_LOG_v1 {"time_micros": 1732001278020624, "job": 264, "event": "flush_started", "num_memtables": 1, "num_entries": 1277483, "num_deletes": 0, "total_data_size": 461309749, "memory_usage": 541326728, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:27:58.020638 139850957195008 [flush_job.cc:910] [default] [JOB 264] Level-0 flush table #5427: started 2024/11/19-15:27:58.021731 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000034.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/005427.sst in ConvertToSST, mem_size = 516.249 M, time = 1079.731 us 2024/11/19-15:27:58.243722 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-02 2024/11/19-15:27:58.243797 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00257/att-02/compact.done) = No such file or directory: 5.145 sec on http probe failed: 4, retry 2024/11/19-15:27:58.263162 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/005427.sst): fsize = 516.251 M, time(ms): open: 0.049, seek: 0.017, finish: 0.047, meta: 0.014, Flush: 0.018, sync: 241.242, close: 0.022, all: 241.409 2024/11/19-15:27:58.264353 139850957195008 [flush_job.cc:1083] [default] [JOB 264] Flush lasted 243762 microseconds, and 189851 cpu microseconds. 2024/11/19-15:27:58.286948 139850957195008 (Original Log Time 2024/11/19-15:27:58.263207) [flush_job.cc:1029] [default] [JOB 264] Level-0 flush table #5427: 541328154 bytes OK (needs compaction) 2024/11/19-15:27:58.286959 139850957195008 (Original Log Time 2024/11/19-15:27:58.264370) [memtable_list.cc:560] [default] Level-0 commit table #5427 started 2024/11/19-15:27:58.286962 139850957195008 (Original Log Time 2024/11/19-15:27:58.286865) [memtable_list.cc:759] [default] Level-0 commit table #5427: memtable #1 done 2024/11/19-15:27:58.286965 139850957195008 (Original Log Time 2024/11/19-15:27:58.286890) EVENT_LOG_v1 {"time_micros": 1732001278286885, "job": 264, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [2, 156, 53, 82, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:27:58.286968 139850957195008 (Original Log Time 2024/11/19-15:27:58.286912) [db_impl_compaction_flush.cc:310] [default] Level summary: files[2 156 53 82 0 0 0] max score 7.96 (184 files need compaction) 2024/11/19-15:27:58.338411 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-02 2024/11/19-15:27:58.338492 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00255/att-02/compact.done) = No such file or directory: 9.217 sec on http probe failed: 4, retry 2024/11/19-15:27:58.429542 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-00 2024/11/19-15:27:58.429628 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00263/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:27:58.782842 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-02 2024/11/19-15:27:58.782935 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00254/att-02/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:27:58.782955 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00254/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.234 sec, real 11.504 (submit 288.249 ms wait 11.208 s), input 127.234 MiB, send shutdown 2024/11/19-15:27:58.792418 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00254/att-02 2024/11/19-15:27:58.792665 139851179493120 [dcompact_etcd.cc:1773] job-00254/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:58.792989 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00254) = success 2024/11/19-15:27:58.792995 139851179493120 [dcompact_etcd.cc:1773] job-00254/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:58.793013 139851179493120 (Original Log Time 2024/11/19-15:27:24.145589) EVENT_LOG_v1 {"time_micros": 1732001244145582, "job": 254, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5256, 5258, 5261, 5264], "score": 9.64524, "input_data_size": 134026368, "oldest_snapshot_seqno": -1} 2024/11/19-15:27:58.793021 139851179493120 [compaction_job.cc:2437] [default] [JOB 254] Compacting 4@1 files to L2, score 9.65, subcompactions 1 : 1 2024/11/19-15:27:58.793029 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 252 Base level 1, inputs: [5256(31MB) 5258(31MB) 5261(31MB) 5264(31MB)] 2024/11/19-15:27:58.793304 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005428.sst) /storage/mytopling/..//tmp/Topling-82038-L6bUET 2024/11/19-15:27:58.853677 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005428.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:27:58.859030 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005428.sst): index pass time = 0.01's, 226.176'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 100277 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:27:58.924356 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005428.sst): first pass time = 0.13's, 512.135'MB/sec 2024/11/19-15:27:58.924409 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005428.sst): old prealloc_size = 73819750, real_size = 65223760 2024/11/19-15:27:58.924479 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005428.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 100277 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0650 GB avg-val = 645.24 avg-zval = 648.49 2024/11/19-15:27:58.924559 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005428.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.071 sec, Key+Value = 0.067 GB 2024/11/19-15:27:58.994338 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005428.sst): second pass time = 0.00's,128377575.397'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 932.597'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2504.815'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.551'MB/sec write SST data time = 0.02's, 2680.496'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 128377575.397'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 100277 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.24 avg-zval = 648.49 usrkeys = 100277 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.24 avg-zval = 648.49 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7942 UnZipSize{ index = 0.0016 GB value = 0.0647 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0002 GB value = 0.0650 GB dict = 0.00 MB all = 0.0652 GB } UnZip/Zip{ index = 9.4316 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 18.729698 GB avg = 0.274 KB (by entry num) total key len = 1.752482 GB avg = 0.026 KB total ukey len = 1.206328 GB avg = 0.018 KB total ukey num = 0.068269259 Billion total entry num = 0.068269259 Billion write speed all = 36.443464552 MB/sec (with seq num) write speed all = 35.471705398 MB/sec (without seq num) 2024/11/19-15:27:58.998371 139851179493120 [compaction_job.cc:1948] [default] [JOB 254] Generated table #5428: 100277 keys, 65200589 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:58.998422 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001278998399, "cf_name": "default", "job": 254, "event": "table_file_creation", "file_number": 5428, "file_size": 65200589, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 65028416, "index_size": 170112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2406648, "raw_average_key_size": 24, "raw_value_size": 64702298, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 100277, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001278, "oldest_key_time": 0, "file_creation_time": 1732001278, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5428, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:59.014786 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005429.sst) /storage/mytopling/..//tmp/Topling-82038-FiFy7F 2024/11/19-15:27:59.072815 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005429.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:27:59.081080 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005429.sst): index pass time = 0.01's, 144.587'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 99111 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:27:59.144575 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005429.sst): first pass time = 0.13's, 510.949'MB/sec 2024/11/19-15:27:59.144632 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005429.sst): old prealloc_size = 73819750, real_size = 64442752 2024/11/19-15:27:59.144707 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005429.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 99111 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0639 GB zip-val = 0.0642 GB avg-val = 645.01 avg-zval = 648.26 2024/11/19-15:27:59.144792 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005429.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.072 sec, Key+Value = 0.066 GB 2024/11/19-15:27:59.214427 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005429.sst): second pass time = 0.00's,230784335.740'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 923.164'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2368.568'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.544'MB/sec write SST data time = 0.02's, 2663.226'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.064'GB zip my value throughput = 230784335.740'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 99111 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.01 avg-zval = 648.26 usrkeys = 99111 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.01 avg-zval = 648.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7847 UnZipSize{ index = 0.0016 GB value = 0.0639 GB dict = 0.00 MB all = 0.0663 GB } __ZipSize{ index = 0.0002 GB value = 0.0642 GB dict = 0.00 MB all = 0.0644 GB } UnZip/Zip{ index = 9.4284 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1061 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 18.793626 GB avg = 0.275 KB (by entry num) total key len = 1.754861 GB avg = 0.026 KB total ukey len = 1.207914 GB avg = 0.018 KB total ukey num = 0.068368370 Billion total entry num = 0.068368370 Billion write speed all = 36.547129132 MB/sec (with seq num) write speed all = 35.574340158 MB/sec (without seq num) 2024/11/19-15:27:59.219056 139851179493120 [compaction_job.cc:1948] [default] [JOB 254] Generated table #5429: 99111 keys, 64419853 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:59.219105 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001279219084, "cf_name": "default", "job": 254, "event": "table_file_creation", "file_number": 5429, "file_size": 64419853, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64249584, "index_size": 168192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2378664, "raw_average_key_size": 24, "raw_value_size": 63927261, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 99111, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001279, "oldest_key_time": 0, "file_creation_time": 1732001279, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5429, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:59.234892 139851179493120 [compaction_job.cc:703] job-00254: subcompact[0], size: 0.129620 G, files: /storage/mytopling/.rocksdb [005428,005429] 2024/11/19-15:27:59.235909 139851179493120 (Original Log Time 2024/11/19-15:27:58.793056) EVENT_LOG_v1 {"time_micros": 1732001278793041, "job": 254, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5256, 5258, 5261, 5264], "score": 9.64524, "input_data_size": 134026368, "oldest_snapshot_seqno": -1} 2024/11/19-15:27:59.235931 139851179493120 (Original Log Time 2024/11/19-15:27:59.235395) [compaction_job.cc:2022] [default] [JOB 254] Compacted 4@1 files to L2 => 129620442 bytes 2024/11/19-15:27:59.235935 139851179493120 (Original Log Time 2024/11/19-15:27:59.235757) [compaction_job.cc:1224] [default] compacted to: files[2 152 55 82 0 0 0] max score 7.96 (186 files need compaction), MB/sec: 303.3 rd, 293.4 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.8, 0.0 +0.0 blob) out(123.6 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 199388, records dropped: 0 output_compression: Snappy 2024/11/19-15:27:59.235955 139851179493120 (Original Log Time 2024/11/19-15:27:59.235785) EVENT_LOG_v1 {"time_micros": 1732001279235772, "cf": "default", "job": 254, "event": "compaction_finished", "compaction_time_micros": 441851, "compaction_time_cpu_micros": 439884, "output_level": 2, "num_output_files": 2, "total_output_size": 129620442, "num_input_records": 199388, "num_output_records": 199388, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 152, 55, 82, 0, 0, 0]} 2024/11/19-15:27:59.238338 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004617.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:59.238363 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001279238360, "job": 254, "event": "table_file_deletion", "file_number": 4617} 2024/11/19-15:27:59.240957 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004619.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:59.240980 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001279240977, "job": 254, "event": "table_file_deletion", "file_number": 4619} 2024/11/19-15:27:59.243881 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004631.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:59.243901 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001279243900, "job": 254, "event": "table_file_deletion", "file_number": 4631} 2024/11/19-15:27:59.249575 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004787.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:27:59.249595 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001279249594, "job": 254, "event": "table_file_deletion", "file_number": 4787} 2024/11/19-15:27:59.249758 139851179493120 [compaction_job.cc:2437] [default] [JOB 265] Compacting 4@1 files to L2, score 7.96, subcompactions 1 : 1 2024/11/19-15:27:59.249767 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 263 Base level 1, inputs: [5323(31MB) 5326(31MB) 5329(31MB) 5333(31MB)] 2024/11/19-15:27:59.312472 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-00 2024/11/19-15:27:59.312565 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00261/att-00/compact.done) = No such file or directory: 9.226 sec on http probe failed: 4, retry 2024/11/19-15:27:59.453703 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-02 2024/11/19-15:27:59.453775 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00257/att-02/compact.done) = No such file or directory: 6.356 sec on http probe failed: 5, giveup 2024/11/19-15:27:59.453792 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00257/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 60.252 sec, real 6.636 (submit 275.757 ms wait 6.356 s), input 60.252 MiB, send shutdown 2024/11/19-15:27:59.462388 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00257/att-02 2024/11/19-15:27:59.462591 139851137550080 [dcompact_etcd.cc:1773] job-00257/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:59.462814 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00257) = success 2024/11/19-15:27:59.462819 139851137550080 [dcompact_etcd.cc:1773] job-00257/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:27:59.462836 139851137550080 (Original Log Time 2024/11/19-15:27:39.453223) EVENT_LOG_v1 {"time_micros": 1732001259453212, "job": 257, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5288, 5289, 5251], "score": 8.89631, "input_data_size": 66366150, "oldest_snapshot_seqno": 152377010} 2024/11/19-15:27:59.462843 139851137550080 [compaction_job.cc:2437] [default] [JOB 257] Compacting 3@1 files to L2, score 8.90, subcompactions 1 : 1 2024/11/19-15:27:59.462850 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 255 Base level 1, inputs: [5288(13MB) 5289(25MB) 5251(24MB)] 2024/11/19-15:27:59.463105 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005430.sst) /storage/mytopling/..//tmp/Topling-82038-NSuHtB 2024/11/19-15:27:59.476025 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005430.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0002 GB, waited 0.000 sec, Key+Value = 0.014 GB 2024/11/19-15:27:59.477285 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005430.sst): index pass time = 0.00's, 201.515'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 20791 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0002 GB zip-key = 0.0000 GB avg-key = 12.00 avg-zkey = 1.76 2024/11/19-15:27:59.491439 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005430.sst): first pass time = 0.03's, 490.795'MB/sec 2024/11/19-15:27:59.491478 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005430.sst): old prealloc_size = 73002765, real_size = 13505904 2024/11/19-15:27:59.491512 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005430.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 20791 prefix = 4 raw-key = 0.0002 GB zip-key = 0.0000 GB avg-key = 12.00 avg-zkey = 1.76 raw-val = 0.0134 GB zip-val = 0.0135 GB avg-val = 644.58 avg-zval = 647.59 2024/11/19-15:27:59.491582 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005430.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0000 GB, waited 0.016 sec, Key+Value = 0.014 GB 2024/11/19-15:27:59.505337 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005430.sst): second pass time = 0.00's,58266921.739'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.01's, 978.975'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1159.464'MB/sec (index lex order gen) rebuild zvType time = 0.01's, 0.589'MB/sec write SST data time = 0.00's, 2720.891'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.013'GB zip my value throughput = 58266921.739'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 20791 avg-key = 24.00 avg-zkey = 1.76 avg-val = 644.58 avg-zval = 647.59 usrkeys = 20791 avg-key = 16.00 avg-zkey = 1.76 avg-val = 651.58 avg-zval = 647.59 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 20791 bytes = 49392 } seq expand size = 145537 multi value expand size = 0 cnt WriteAppend = 1652 UnZipSize{ index = 0.0003 GB value = 0.0134 GB dict = 0.00 MB all = 0.0139 GB } __ZipSize{ index = 0.0000 GB value = 0.0135 GB dict = 0.00 MB all = 0.0136 GB } UnZip/Zip{ index = 9.1029 value = 0.9954 dict = 0.00 all = 1.0258 } Zip/UnZip{ index = 0.1099 value = 1.0047 dict = 0.00 all = 0.9749 } ---------------------------- total value len = 18.807172 GB avg = 0.275 KB (by entry num) total key len = 1.755360 GB avg = 0.026 KB total ukey len = 1.208247 GB avg = 0.018 KB total ukey num = 0.068389161 Billion total entry num = 0.068389161 Billion write speed all = 36.553197910 MB/sec (with seq num) write speed all = 35.580616333 MB/sec (without seq num) 2024/11/19-15:27:59.506627 139851137550080 [compaction_job.cc:1948] [default] [JOB 257] Generated table #5430: 20791 keys, 13551385 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:59.506678 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001279506654, "cf_name": "default", "job": 257, "event": "table_file_creation", "file_number": 5430, "file_size": 13551385, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 150453639, "largest_seqno": 150887069, "table_properties": {"data_size": 13463968, "index_size": 36544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 498984, "raw_average_key_size": 24, "raw_value_size": 13401392, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 20791, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001279, "oldest_key_time": 0, "file_creation_time": 1732001279, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5430, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:59.510210 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005431.sst) /storage/mytopling/..//tmp/Topling-82038-uEkU26 2024/11/19-15:27:59.542682 139851179493120 [dcompact_etcd.cc:1163] job-00265/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.292 sec, input = 127.818 MiB 2024/11/19-15:27:59.689956 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005431.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.049 GB 2024/11/19-15:27:59.697703 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005431.sst): index pass time = 0.01's, 690.531'MB/sec index type = UintIndex_IL_256_32 usrkeys = 666001 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0053 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 0.16 2024/11/19-15:27:59.712680 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-00 2024/11/19-15:27:59.712747 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00262/att-00/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:27:59.742044 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005431.sst): first pass time = 0.23's, 212.575'MB/sec 2024/11/19-15:27:59.742097 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005431.sst): old prealloc_size = 73002765, real_size = 38394640 2024/11/19-15:27:59.742130 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005431.sst): index type = UintIndex_IL_256_32 , store type = PlainBlobStore usrkeys = 666001 prefix = 4 raw-key = 0.0053 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 0.16 raw-val = 0.0360 GB zip-val = 0.0381 GB avg-val = 53.99 avg-zval = 57.24 2024/11/19-15:27:59.829294 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005431.sst): second pass time = 0.00's,153670602.564'MB/sec, value only(73.0% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 461.406'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 4401.688'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 4.077'MB/sec write SST data time = 0.05's, 868.440'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.036'GB zip my value throughput = 153670602.564'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 666001 avg-key = 20.00 avg-zkey = 0.16 avg-val = 53.99 avg-zval = 57.24 usrkeys = 666001 avg-key = 12.00 avg-zkey = 0.16 avg-val = 60.99 avg-zval = 57.24 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 666001 bytes = 1998016 } seq expand size = 4662007 multi value expand size = 0 cnt WriteAppend = 4659 UnZipSize{ index = 0.0080 GB value = 0.0360 GB dict = 0.00 MB all = 0.0493 GB } __ZipSize{ index = 0.0001 GB value = 0.0381 GB dict = 0.00 MB all = 0.0402 GB } UnZip/Zip{ index = 76.6105 value = 0.9432 dict = 0.00 all = 1.2250 } Zip/UnZip{ index = 0.0131 value = 1.0602 dict = 0.00 all = 0.8163 } ---------------------------- total value len = 18.847793 GB avg = 0.273 KB (by entry num) total key len = 1.768680 GB avg = 0.026 KB total ukey len = 1.216239 GB avg = 0.018 KB total ukey num = 0.069055162 Billion total entry num = 0.069055162 Billion write speed all = 36.627993223 MB/sec (with seq num) write speed all = 35.646505477 MB/sec (without seq num) 2024/11/19-15:27:59.831714 139851137550080 [compaction_job.cc:1948] [default] [JOB 257] Generated table #5431: 666001 keys, 40227161 bytes (need compaction), temperature: kUnknown 2024/11/19-15:27:59.831762 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001279831740, "cf_name": "default", "job": 257, "event": "table_file_creation", "file_number": 5431, "file_size": 40227161, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 139817829, "largest_seqno": 150820309, "table_properties": {"data_size": 38123648, "index_size": 104320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13320020, "raw_average_key_size": 20, "raw_value_size": 35958921, "raw_average_value_size": 53, "num_data_blocks": 1, "num_entries": 666001, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-UintIndex_IL_256_32-PlainBlobStore", "creation_time": 1732001279, "oldest_key_time": 0, "file_creation_time": 1732001279, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5431, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:27:59.841817 139851137550080 [compaction_job.cc:703] job-00257: subcompact[0], size: 0.053779 G, files: /storage/mytopling/.rocksdb [005430,005431] 2024/11/19-15:27:59.842805 139851137550080 (Original Log Time 2024/11/19-15:27:59.462876) EVENT_LOG_v1 {"time_micros": 1732001279462862, "job": 257, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5288, 5289, 5251], "score": 8.89631, "input_data_size": 66366150, "oldest_snapshot_seqno": 152377010} 2024/11/19-15:27:59.842814 139851137550080 (Original Log Time 2024/11/19-15:27:59.842329) [compaction_job.cc:2022] [default] [JOB 257] Compacted 3@1 files to L2 => 53778546 bytes 2024/11/19-15:27:59.842817 139851137550080 (Original Log Time 2024/11/19-15:27:59.842684) [compaction_job.cc:1224] [default] compacted to: files[2 149 57 82 0 0 0] max score 7.71 (184 files need compaction), MB/sec: 175.1 rd, 141.9 wr, level 2, files in(3, 0) out(2 +0 blob) MB in(63.3, 0.0 +0.0 blob) out(51.3 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 686792, records dropped: 0 output_compression: Snappy 2024/11/19-15:27:59.842833 139851137550080 (Original Log Time 2024/11/19-15:27:59.842706) EVENT_LOG_v1 {"time_micros": 1732001279842694, "cf": "default", "job": 257, "event": "compaction_finished", "compaction_time_micros": 378953, "compaction_time_cpu_micros": 375942, "output_level": 2, "num_output_files": 2, "total_output_size": 53778546, "num_input_records": 686792, "num_output_records": 686792, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 149, 57, 82, 0, 0, 0]} 2024/11/19-15:27:59.842909 139851137550080 [compaction_job.cc:2437] [default] [JOB 266] Compacting 4@1 files to L2, score 7.71, subcompactions 1 : 1 2024/11/19-15:27:59.842918 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 264 Base level 1, inputs: [5336(31MB) 5339(31MB) 5342(31MB) 5344(31MB)] 2024/11/19-15:28:00.136339 139851137550080 [dcompact_etcd.cc:1163] job-00266/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.293 sec, input = 127.820 MiB 2024/11/19-15:28:00.346514 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-02 2024/11/19-15:28:00.346592 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00255/att-02/compact.done) = No such file or directory: 11.227 sec on http probe failed: 5, giveup 2024/11/19-15:28:00.346610 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00255/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.240 sec, real 11.532 (submit 297.719 ms wait 11.227 s), input 127.240 MiB, send shutdown 2024/11/19-15:28:00.355796 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00255/att-02 2024/11/19-15:28:00.356009 139851127064320 [dcompact_etcd.cc:1773] job-00255/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:00.356238 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00255) = success 2024/11/19-15:28:00.356244 139851127064320 [dcompact_etcd.cc:1773] job-00255/att-03: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:00.356261 139851127064320 (Original Log Time 2024/11/19-15:27:25.656204) EVENT_LOG_v1 {"time_micros": 1732001245656196, "job": 255, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5267, 5269, 5271, 5273], "score": 9.3956, "input_data_size": 134027520, "oldest_snapshot_seqno": 151331766} 2024/11/19-15:28:00.356270 139851127064320 [compaction_job.cc:2437] [default] [JOB 255] Compacting 4@1 files to L2, score 9.40, subcompactions 1 : 1 2024/11/19-15:28:00.356278 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 253 Base level 1, inputs: [5267(31MB) 5269(31MB) 5271(31MB) 5273(31MB)] 2024/11/19-15:28:00.356528 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005432.sst) /storage/mytopling/..//tmp/Topling-82038-Q8R61r 2024/11/19-15:28:00.418144 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005432.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:28:00.423480 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005432.sst): index pass time = 0.01's, 227.288'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 100417 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:28:00.437456 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-00 2024/11/19-15:28:00.437530 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00263/att-00/compact.done) = No such file or directory: 5.204 sec on http probe failed: 2, retry 2024/11/19-15:28:00.494396 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005432.sst): first pass time = 0.14's, 486.819'MB/sec 2024/11/19-15:28:00.494454 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005432.sst): old prealloc_size = 73819750, real_size = 65221664 2024/11/19-15:28:00.494525 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005432.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 100417 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0650 GB avg-val = 644.31 avg-zval = 647.56 2024/11/19-15:28:00.494610 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005432.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.076 sec, Key+Value = 0.067 GB 2024/11/19-15:28:00.562780 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005432.sst): second pass time = 0.00's,210747355.049'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 954.432'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2554.588'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.562'MB/sec write SST data time = 0.02's, 2765.775'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 210747355.049'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 100417 avg-key = 24.00 avg-zkey = 1.70 avg-val = 644.31 avg-zval = 647.56 usrkeys = 100417 avg-key = 16.00 avg-zkey = 1.70 avg-val = 644.31 avg-zval = 647.56 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7942 UnZipSize{ index = 0.0016 GB value = 0.0647 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0002 GB value = 0.0650 GB dict = 0.00 MB all = 0.0652 GB } UnZip/Zip{ index = 9.4306 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9715 } ---------------------------- total value len = 18.912493 GB avg = 0.273 KB (by entry num) total key len = 1.771090 GB avg = 0.026 KB total ukey len = 1.217845 GB avg = 0.018 KB total ukey num = 0.069155579 Billion total entry num = 0.069155579 Billion write speed all = 36.699398225 MB/sec (with seq num) write speed all = 35.717762444 MB/sec (without seq num) 2024/11/19-15:28:00.566796 139851127064320 [compaction_job.cc:1948] [default] [JOB 255] Generated table #5432: 100417 keys, 65198477 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:00.566848 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001280566825, "cf_name": "default", "job": 255, "event": "table_file_creation", "file_number": 5432, "file_size": 65198477, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 65026000, "index_size": 170368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2410008, "raw_average_key_size": 24, "raw_value_size": 64699438, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 100417, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001280, "oldest_key_time": 0, "file_creation_time": 1732001280, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5432, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:00.582407 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005433.sst) /storage/mytopling/..//tmp/Topling-82038-J6Hjlt 2024/11/19-15:28:00.642272 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005433.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:28:00.647647 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005433.sst): index pass time = 0.01's, 222.630'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 99105 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:28:00.711269 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005433.sst): first pass time = 0.13's, 514.659'MB/sec 2024/11/19-15:28:00.711324 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005433.sst): old prealloc_size = 73819750, real_size = 64447888 2024/11/19-15:28:00.711396 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005433.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 99105 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0639 GB zip-val = 0.0643 GB avg-val = 645.10 avg-zval = 648.35 2024/11/19-15:28:00.711478 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005433.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.069 sec, Key+Value = 0.066 GB 2024/11/19-15:28:00.779444 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005433.sst): second pass time = 0.00's,297360069.767'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 945.979'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2448.637'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.558'MB/sec write SST data time = 0.02's, 2725.206'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.064'GB zip my value throughput = 297360069.767'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 99105 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.10 avg-zval = 648.35 usrkeys = 99105 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.10 avg-zval = 648.35 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7847 UnZipSize{ index = 0.0016 GB value = 0.0639 GB dict = 0.00 MB all = 0.0663 GB } __ZipSize{ index = 0.0002 GB value = 0.0643 GB dict = 0.00 MB all = 0.0644 GB } UnZip/Zip{ index = 9.4278 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1061 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 18.976425 GB avg = 0.274 KB (by entry num) total key len = 1.773469 GB avg = 0.026 KB total ukey len = 1.219431 GB avg = 0.018 KB total ukey num = 0.069254684 Billion total entry num = 0.069254684 Billion write speed all = 36.802907111 MB/sec (with seq num) write speed all = 35.820242343 MB/sec (without seq num) 2024/11/19-15:28:00.783868 139851127064320 [compaction_job.cc:1948] [default] [JOB 255] Generated table #5433: 99105 keys, 64424973 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:00.783931 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001280783898, "cf_name": "default", "job": 255, "event": "table_file_creation", "file_number": 5433, "file_size": 64424973, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64254720, "index_size": 168192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2378520, "raw_average_key_size": 24, "raw_value_size": 63932415, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 99105, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001280, "oldest_key_time": 0, "file_creation_time": 1732001280, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5433, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:00.799599 139851127064320 [compaction_job.cc:703] job-00255: subcompact[0], size: 0.129623 G, files: /storage/mytopling/.rocksdb [005432,005433] 2024/11/19-15:28:00.800593 139851127064320 (Original Log Time 2024/11/19-15:28:00.356305) EVENT_LOG_v1 {"time_micros": 1732001280356290, "job": 255, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5267, 5269, 5271, 5273], "score": 9.3956, "input_data_size": 134027520, "oldest_snapshot_seqno": 151331766} 2024/11/19-15:28:00.800604 139851127064320 (Original Log Time 2024/11/19-15:28:00.800089) [compaction_job.cc:2022] [default] [JOB 255] Compacted 4@1 files to L2 => 129623450 bytes 2024/11/19-15:28:00.800608 139851127064320 (Original Log Time 2024/11/19-15:28:00.800462) [compaction_job.cc:1224] [default] compacted to: files[2 145 59 82 0 0 0] max score 7.46 (182 files need compaction), MB/sec: 302.3 rd, 292.4 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.8, 0.0 +0.0 blob) out(123.6 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 199522, records dropped: 0 output_compression: Snappy 2024/11/19-15:28:00.800628 139851127064320 (Original Log Time 2024/11/19-15:28:00.800488) EVENT_LOG_v1 {"time_micros": 1732001280800475, "cf": "default", "job": 255, "event": "compaction_finished", "compaction_time_micros": 443309, "compaction_time_cpu_micros": 441012, "output_level": 2, "num_output_files": 2, "total_output_size": 129623450, "num_input_records": 199522, "num_output_records": 199522, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 145, 59, 82, 0, 0, 0]} 2024/11/19-15:28:00.803186 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004630.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:00.803211 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001280803209, "job": 255, "event": "table_file_deletion", "file_number": 4630} 2024/11/19-15:28:00.808139 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004792.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:00.808161 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001280808158, "job": 255, "event": "table_file_deletion", "file_number": 4792} 2024/11/19-15:28:00.812915 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004793.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:00.812944 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001280812942, "job": 255, "event": "table_file_deletion", "file_number": 4793} 2024/11/19-15:28:00.817945 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004796.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:00.817966 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001280817964, "job": 255, "event": "table_file_deletion", "file_number": 4796} 2024/11/19-15:28:00.818566 139851127064320 [compaction_job.cc:2437] [default] [JOB 267] Compacting 4@1 files to L2, score 7.46, subcompactions 1 : 1 2024/11/19-15:28:00.818581 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 265 Base level 1, inputs: [5346(31MB) 5350(31MB) 5355(31MB) 5359(31MB)] 2024/11/19-15:28:01.132729 139851127064320 [dcompact_etcd.cc:1163] job-00267/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.314 sec, input = 127.822 MiB 2024/11/19-15:28:01.207883 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-02 2024/11/19-15:28:01.207974 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00256/att-02/compact.done) = No such file or directory: 3.181 sec on http probe failed: 1, retry 2024/11/19-15:28:01.321491 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-00 2024/11/19-15:28:01.321588 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00261/att-00/compact.done) = No such file or directory: 11.232 sec on http probe failed: 5, giveup 2024/11/19-15:28:01.321606 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00261/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.819 sec, real 11.514 (submit 272.853 ms wait 11.232 s), input 127.819 MiB, send shutdown 2024/11/19-15:28:01.330685 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-00 2024/11/19-15:28:01.330916 139851148035840 [dcompact_etcd.cc:1773] job-00261/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:01.637211 139851148035840 [dcompact_etcd.cc:1163] job-00261/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.306 sec, input = 127.819 MiB 2024/11/19-15:28:01.721386 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-00 2024/11/19-15:28:01.721485 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00262/att-00/compact.done) = No such file or directory: 7.211 sec on http probe failed: 3, retry 2024/11/19-15:28:02.446420 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-00 2024/11/19-15:28:02.446532 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00263/att-00/compact.done) = No such file or directory: 7.212 sec on http probe failed: 3, retry 2024/11/19-15:28:02.747237 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-00 2024/11/19-15:28:02.747322 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00265/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:03.213439 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-02 2024/11/19-15:28:03.213515 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00256/att-02/compact.done) = No such file or directory: 5.190 sec on http probe failed: 2, retry 2024/11/19-15:28:03.340631 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-00 2024/11/19-15:28:03.340746 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00266/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:03.730460 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-00 2024/11/19-15:28:03.730533 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00262/att-00/compact.done) = No such file or directory: 9.220 sec on http probe failed: 4, retry 2024/11/19-15:28:04.335721 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-00 2024/11/19-15:28:04.335833 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00267/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:04.454612 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-00 2024/11/19-15:28:04.454706 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00263/att-00/compact.done) = No such file or directory: 9.221 sec on http probe failed: 4, retry 2024/11/19-15:28:04.752477 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-00 2024/11/19-15:28:04.752549 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00265/att-00/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:28:04.841323 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-01 2024/11/19-15:28:04.841394 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00261/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:05.220489 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-02 2024/11/19-15:28:05.220576 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00256/att-02/compact.done) = No such file or directory: 7.195 sec on http probe failed: 3, retry 2024/11/19-15:28:05.347652 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-00 2024/11/19-15:28:05.347728 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00266/att-00/compact.done) = No such file or directory: 5.204 sec on http probe failed: 2, retry 2024/11/19-15:28:05.736301 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-00 2024/11/19-15:28:05.736399 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00262/att-00/compact.done) = No such file or directory: 11.229 sec on http probe failed: 5, giveup 2024/11/19-15:28:05.736417 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00262/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.819 sec, real 11.533 (submit 298.736 ms wait 11.229 s), input 127.819 MiB, send shutdown 2024/11/19-15:28:05.740950 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-00 2024/11/19-15:28:05.741170 139851158521600 [dcompact_etcd.cc:1773] job-00262/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:06.019593 139851158521600 [dcompact_etcd.cc:1163] job-00262/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.278 sec, input = 127.819 MiB 2024/11/19-15:28:06.344977 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-00 2024/11/19-15:28:06.345058 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00267/att-00/compact.done) = No such file or directory: 5.203 sec on http probe failed: 2, retry 2024/11/19-15:28:06.463015 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-00 2024/11/19-15:28:06.463118 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00263/att-00/compact.done) = No such file or directory: 11.229 sec on http probe failed: 5, giveup 2024/11/19-15:28:06.463136 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00263/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.818 sec, real 11.511 (submit 272.943 ms wait 11.229 s), input 127.818 MiB, send shutdown 2024/11/19-15:28:06.467564 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-00 2024/11/19-15:28:06.467782 139850978166528 [dcompact_etcd.cc:1773] job-00263/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:06.751839 139850978166528 [dcompact_etcd.cc:1163] job-00263/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.284 sec, input = 127.818 MiB 2024/11/19-15:28:06.757865 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-00 2024/11/19-15:28:06.757927 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00265/att-00/compact.done) = No such file or directory: 7.210 sec on http probe failed: 3, retry 2024/11/19-15:28:06.846538 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-01 2024/11/19-15:28:06.846649 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00261/att-01/compact.done) = No such file or directory: 5.204 sec on http probe failed: 2, retry 2024/11/19-15:28:07.226133 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-02 2024/11/19-15:28:07.226206 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00256/att-02/compact.done) = No such file or directory: 9.202 sec on http probe failed: 4, retry 2024/11/19-15:28:07.352495 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-00 2024/11/19-15:28:07.352573 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00266/att-00/compact.done) = No such file or directory: 7.211 sec on http probe failed: 3, retry 2024/11/19-15:28:08.353221 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-00 2024/11/19-15:28:08.353315 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00267/att-00/compact.done) = No such file or directory: 7.212 sec on http probe failed: 3, retry 2024/11/19-15:28:08.763119 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-00 2024/11/19-15:28:08.763207 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00265/att-00/compact.done) = No such file or directory: 9.215 sec on http probe failed: 4, retry 2024/11/19-15:28:08.851859 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-01 2024/11/19-15:28:08.851944 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00261/att-01/compact.done) = No such file or directory: 7.210 sec on http probe failed: 3, retry 2024/11/19-15:28:09.224324 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-01 2024/11/19-15:28:09.224426 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00262/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:09.235090 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-02 2024/11/19-15:28:09.235165 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00256/att-02/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:28:09.235183 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00256/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.238 sec, real 11.522 (submit 305.696 ms wait 11.208 s), input 127.238 MiB, send shutdown 2024/11/19-15:28:09.243895 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00256/att-02 2024/11/19-15:28:09.244108 139851169007360 [dcompact_etcd.cc:1773] job-00256/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:09.246032 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00256) = success 2024/11/19-15:28:09.246041 139851169007360 [dcompact_etcd.cc:1773] job-00256/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:28:09.246058 139851169007360 (Original Log Time 2024/11/19-15:27:34.677779) EVENT_LOG_v1 {"time_micros": 1732001254677770, "job": 256, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5276, 5279, 5282, 5284], "score": 9.14596, "input_data_size": 134027968, "oldest_snapshot_seqno": 151977464} 2024/11/19-15:28:09.246065 139851169007360 [compaction_job.cc:2437] [default] [JOB 256] Compacting 4@1 files to L2, score 9.15, subcompactions 1 : 1 2024/11/19-15:28:09.246073 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 254 Base level 1, inputs: [5276(31MB) 5279(31MB) 5282(31MB) 5284(31MB)] 2024/11/19-15:28:09.246285 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005434.sst) /storage/mytopling/..//tmp/Topling-82038-cY1Tek 2024/11/19-15:28:09.306204 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005434.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:28:09.311620 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005434.sst): index pass time = 0.01's, 223.711'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 100268 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:28:09.360741 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-00 2024/11/19-15:28:09.360842 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00266/att-00/compact.done) = No such file or directory: 9.216 sec on http probe failed: 4, retry 2024/11/19-15:28:09.374428 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005434.sst): first pass time = 0.13's, 523.752'MB/sec 2024/11/19-15:28:09.374478 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005434.sst): old prealloc_size = 73819750, real_size = 65223888 2024/11/19-15:28:09.374551 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005434.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 100268 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0650 GB avg-val = 645.30 avg-zval = 648.55 2024/11/19-15:28:09.374605 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005434.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.068 sec, Key+Value = 0.067 GB 2024/11/19-15:28:09.442146 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005434.sst): second pass time = 0.00's,220076608.844'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 963.749'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2419.820'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.564'MB/sec write SST data time = 0.02's, 2815.894'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 220076608.844'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 100268 avg-key = 24.00 avg-zkey = 1.70 avg-val = 645.30 avg-zval = 648.55 usrkeys = 100268 avg-key = 16.00 avg-zkey = 1.70 avg-val = 645.30 avg-zval = 648.55 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7942 UnZipSize{ index = 0.0016 GB value = 0.0647 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0002 GB value = 0.0650 GB dict = 0.00 MB all = 0.0652 GB } UnZip/Zip{ index = 9.4343 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9716 } ---------------------------- total value len = 19.041128 GB avg = 0.275 KB (by entry num) total key len = 1.775875 GB avg = 0.026 KB total ukey len = 1.221035 GB avg = 0.018 KB total ukey num = 0.069354952 Billion total entry num = 0.069354952 Billion write speed all = 36.363230047 MB/sec (with seq num) write speed all = 35.394033818 MB/sec (without seq num) 2024/11/19-15:28:09.446105 139851169007360 [compaction_job.cc:1948] [default] [JOB 256] Generated table #5434: 100268 keys, 65200717 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:09.446157 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001289446134, "cf_name": "default", "job": 256, "event": "table_file_creation", "file_number": 5434, "file_size": 65200717, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 65028608, "index_size": 170048, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2406432, "raw_average_key_size": 24, "raw_value_size": 64702523, "raw_average_value_size": 645, "num_data_blocks": 1, "num_entries": 100268, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001289, "oldest_key_time": 0, "file_creation_time": 1732001289, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5434, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:09.460846 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005435.sst) /storage/mytopling/..//tmp/Topling-82038-jmU65F 2024/11/19-15:28:09.519339 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005435.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:28:09.524644 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005435.sst): index pass time = 0.01's, 225.646'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 99157 min-keylen = 12 max-keylen = 12 prefix = 4 subcplen = 3 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 2024/11/19-15:28:09.585754 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005435.sst): first pass time = 0.12's, 530.948'MB/sec 2024/11/19-15:28:09.585803 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005435.sst): old prealloc_size = 73819750, real_size = 64446144 2024/11/19-15:28:09.585875 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005435.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 99157 prefix = 4 raw-key = 0.0012 GB zip-key = 0.0002 GB avg-key = 12.00 avg-zkey = 1.70 raw-val = 0.0639 GB zip-val = 0.0643 GB avg-val = 644.74 avg-zval = 647.99 2024/11/19-15:28:09.585941 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005435.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.067 sec, Key+Value = 0.066 GB 2024/11/19-15:28:09.652187 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005435.sst): second pass time = 0.00's,295974560.185'MB/sec, value only(96.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 970.666'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2432.383'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.571'MB/sec write SST data time = 0.02's, 2817.209'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.064'GB zip my value throughput = 295974560.185'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 99157 avg-key = 24.00 avg-zkey = 1.70 avg-val = 644.74 avg-zval = 647.99 usrkeys = 99157 avg-key = 16.00 avg-zkey = 1.70 avg-val = 644.74 avg-zval = 647.99 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7848 UnZipSize{ index = 0.0016 GB value = 0.0639 GB dict = 0.00 MB all = 0.0663 GB } __ZipSize{ index = 0.0002 GB value = 0.0643 GB dict = 0.00 MB all = 0.0644 GB } UnZip/Zip{ index = 9.4327 value = 0.9950 dict = 0.00 all = 1.0293 } Zip/UnZip{ index = 0.1060 value = 1.0050 dict = 0.00 all = 0.9715 } ---------------------------- total value len = 19.105058 GB avg = 0.275 KB (by entry num) total key len = 1.778255 GB avg = 0.026 KB total ukey len = 1.222622 GB avg = 0.018 KB total ukey num = 0.069454109 Billion total entry num = 0.069454109 Billion write speed all = 36.465681599 MB/sec (with seq num) write speed all = 35.495455690 MB/sec (without seq num) 2024/11/19-15:28:09.655953 139851169007360 [compaction_job.cc:1948] [default] [JOB 256] Generated table #5435: 99157 keys, 64423245 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:09.656007 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001289655983, "cf_name": "default", "job": 256, "event": "table_file_creation", "file_number": 5435, "file_size": 64423245, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64252976, "index_size": 168192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2379768, "raw_average_key_size": 24, "raw_value_size": 63930505, "raw_average_value_size": 644, "num_data_blocks": 1, "num_entries": 99157, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 24, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001289, "oldest_key_time": 0, "file_creation_time": 1732001289, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5435, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:09.670276 139851169007360 [compaction_job.cc:703] job-00256: subcompact[0], size: 0.129624 G, files: /storage/mytopling/.rocksdb [005434,005435] 2024/11/19-15:28:09.671211 139851169007360 (Original Log Time 2024/11/19-15:28:09.246102) EVENT_LOG_v1 {"time_micros": 1732001289246085, "job": 256, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5276, 5279, 5282, 5284], "score": 9.14596, "input_data_size": 134027968, "oldest_snapshot_seqno": 151977464} 2024/11/19-15:28:09.671221 139851169007360 (Original Log Time 2024/11/19-15:28:09.670721) [compaction_job.cc:2022] [default] [JOB 256] Compacted 4@1 files to L2 => 129623962 bytes 2024/11/19-15:28:09.671224 139851169007360 (Original Log Time 2024/11/19-15:28:09.671092) [compaction_job.cc:1224] [default] compacted to: files[2 141 61 82 0 0 0] max score 7.21 (180 files need compaction), MB/sec: 316.0 rd, 305.6 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.8, 0.0 +0.0 blob) out(123.6 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 199425, records dropped: 0 output_compression: Snappy 2024/11/19-15:28:09.671243 139851169007360 (Original Log Time 2024/11/19-15:28:09.671114) EVENT_LOG_v1 {"time_micros": 1732001289671103, "cf": "default", "job": 256, "event": "compaction_finished", "compaction_time_micros": 424186, "compaction_time_cpu_micros": 420986, "output_level": 2, "num_output_files": 2, "total_output_size": 129623962, "num_input_records": 199425, "num_output_records": 199425, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 141, 61, 82, 0, 0, 0]} 2024/11/19-15:28:09.675681 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004797.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:09.675705 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001289675702, "job": 256, "event": "table_file_deletion", "file_number": 4797} 2024/11/19-15:28:09.680289 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004800.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:09.680310 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001289680308, "job": 256, "event": "table_file_deletion", "file_number": 4800} 2024/11/19-15:28:09.685623 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004801.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:09.685643 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001289685641, "job": 256, "event": "table_file_deletion", "file_number": 4801} 2024/11/19-15:28:09.691064 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004802.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:09.691084 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001289691082, "job": 256, "event": "table_file_deletion", "file_number": 4802} 2024/11/19-15:28:09.696646 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004803.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:09.696666 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001289696664, "job": 256, "event": "table_file_deletion", "file_number": 4803} 2024/11/19-15:28:09.702157 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004805.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:09.702176 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001289702174, "job": 256, "event": "table_file_deletion", "file_number": 4805} 2024/11/19-15:28:09.707581 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004806.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:09.707600 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001289707598, "job": 256, "event": "table_file_deletion", "file_number": 4806} 2024/11/19-15:28:09.713215 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004810.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:09.713234 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001289713232, "job": 256, "event": "table_file_deletion", "file_number": 4810} 2024/11/19-15:28:09.719001 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004811.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:09.719020 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001289719018, "job": 256, "event": "table_file_deletion", "file_number": 4811} 2024/11/19-15:28:09.724229 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004814.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:09.724248 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001289724246, "job": 256, "event": "table_file_deletion", "file_number": 4814} 2024/11/19-15:28:09.730197 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004815.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:09.730215 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001289730213, "job": 256, "event": "table_file_deletion", "file_number": 4815} 2024/11/19-15:28:09.735640 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004980.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:09.735658 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001289735656, "job": 256, "event": "table_file_deletion", "file_number": 4980} 2024/11/19-15:28:09.736467 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005260.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:09.736483 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001289736481, "job": 256, "event": "table_file_deletion", "file_number": 5260} 2024/11/19-15:28:09.736637 139851169007360 [compaction_job.cc:2437] [default] [JOB 268] Compacting 4@1 files to L2, score 7.21, subcompactions 1 : 1 2024/11/19-15:28:09.736646 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 266 Base level 1, inputs: [5364(31MB) 5367(31MB) 5371(31MB) 5374(31MB)] 2024/11/19-15:28:09.955891 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-01 2024/11/19-15:28:09.956007 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00263/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:10.045424 139851169007360 [dcompact_etcd.cc:1163] job-00268/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.308 sec, input = 127.823 MiB 2024/11/19-15:28:10.362187 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-00 2024/11/19-15:28:10.362283 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00267/att-00/compact.done) = No such file or directory: 9.221 sec on http probe failed: 4, retry 2024/11/19-15:28:10.768711 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-00 2024/11/19-15:28:10.768791 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00265/att-00/compact.done) = No such file or directory: 11.221 sec on http probe failed: 5, giveup 2024/11/19-15:28:10.768809 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00265/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.818 sec, real 11.519 (submit 292.464 ms wait 11.221 s), input 127.818 MiB, send shutdown 2024/11/19-15:28:10.777082 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-00 2024/11/19-15:28:10.777306 139851179493120 [dcompact_etcd.cc:1773] job-00265/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:10.859934 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-01 2024/11/19-15:28:10.860048 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00261/att-01/compact.done) = No such file or directory: 9.215 sec on http probe failed: 4, retry 2024/11/19-15:28:11.057472 139851179493120 [dcompact_etcd.cc:1163] job-00265/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 127.818 MiB 2024/11/19-15:28:11.233448 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-01 2024/11/19-15:28:11.233525 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00262/att-01/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:28:11.369367 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-00 2024/11/19-15:28:11.369439 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00266/att-00/compact.done) = No such file or directory: 11.225 sec on http probe failed: 5, giveup 2024/11/19-15:28:11.369458 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00266/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.526 (submit 292.979 ms wait 11.225 s), input 127.820 MiB, send shutdown 2024/11/19-15:28:11.378235 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-00 2024/11/19-15:28:11.378466 139851137550080 [dcompact_etcd.cc:1773] job-00266/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:11.666242 139851137550080 [dcompact_etcd.cc:1163] job-00266/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.288 sec, input = 127.820 MiB 2024/11/19-15:28:11.964705 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-01 2024/11/19-15:28:11.964809 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00263/att-01/compact.done) = No such file or directory: 5.204 sec on http probe failed: 2, retry 2024/11/19-15:28:12.370657 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-00 2024/11/19-15:28:12.370733 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00267/att-00/compact.done) = No such file or directory: 11.230 sec on http probe failed: 5, giveup 2024/11/19-15:28:12.370752 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00267/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.822 sec, real 11.552 (submit 313.677 ms wait 11.230 s), input 127.822 MiB, send shutdown 2024/11/19-15:28:12.375540 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-00 2024/11/19-15:28:12.375755 139851127064320 [dcompact_etcd.cc:1773] job-00267/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:12.646489 139851127064320 [dcompact_etcd.cc:1163] job-00267/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.271 sec, input = 127.822 MiB 2024/11/19-15:28:12.865826 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-01 2024/11/19-15:28:12.865942 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00261/att-01/compact.done) = No such file or directory: 11.223 sec on http probe failed: 5, giveup 2024/11/19-15:28:12.865969 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00261/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.819 sec, real 11.535 (submit 306.209 ms wait 11.223 s), input 127.819 MiB, send shutdown 2024/11/19-15:28:12.874592 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-01 2024/11/19-15:28:12.874856 139851148035840 [dcompact_etcd.cc:1773] job-00261/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:13.164325 139851148035840 [dcompact_etcd.cc:1163] job-00261/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 127.819 MiB 2024/11/19-15:28:13.238680 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-01 2024/11/19-15:28:13.238748 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00262/att-01/compact.done) = No such file or directory: 7.214 sec on http probe failed: 3, retry 2024/11/19-15:28:13.250042 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-00 2024/11/19-15:28:13.250110 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00268/att-00/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:13.974666 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-01 2024/11/19-15:28:13.974745 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00263/att-01/compact.done) = No such file or directory: 7.213 sec on http probe failed: 3, retry 2024/11/19-15:28:14.261011 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-01 2024/11/19-15:28:14.261123 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00265/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:14.868238 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-01 2024/11/19-15:28:14.868323 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00266/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:15.247729 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-01 2024/11/19-15:28:15.247857 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00262/att-01/compact.done) = No such file or directory: 9.219 sec on http probe failed: 4, retry 2024/11/19-15:28:15.258640 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-00 2024/11/19-15:28:15.258711 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00268/att-00/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:28:15.307353 139852362282752 [db_impl_write.cc:2181] [default] New memtable created with log file: #5436. Immutable memtables: 0. 2024/11/19-15:28:15.307833 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 269] Syncing log #5426 2024/11/19-15:28:15.307931 139850967680768 (Original Log Time 2024/11/19-15:28:15.307613) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:28:15.307940 139850967680768 [flush_job.cc:879] [default] [JOB 269] Flushing memtable with next log file: 5436 2024/11/19-15:28:15.307961 139850967680768 EVENT_LOG_v1 {"time_micros": 1732001295307956, "job": 269, "event": "flush_started", "num_memtables": 1, "num_entries": 1326551, "num_deletes": 0, "total_data_size": 505404180, "memory_usage": 536608136, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:28:15.307966 139850967680768 [flush_job.cc:910] [default] [JOB 269] Level-0 flush table #5437: started 2024/11/19-15:28:15.308326 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000035.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/005437.sst in ConvertToSST, mem_size = 511.749 M, time = 352.722 us 2024/11/19-15:28:15.833901 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/005437.sst): fsize = 511.751 M, time(ms): open: 0.029, seek: 0.013, finish: 0.035, meta: 0.013, Flush: 0.012, sync: 525.448, close: 0.011, all: 525.561 2024/11/19-15:28:15.835078 139850967680768 [flush_job.cc:1083] [default] [JOB 269] Flush lasted 527147 microseconds, and 421116 cpu microseconds. 2024/11/19-15:28:15.851009 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-01 2024/11/19-15:28:15.851098 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00267/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:15.858109 139850967680768 (Original Log Time 2024/11/19-15:28:15.833956) [flush_job.cc:1029] [default] [JOB 269] Level-0 flush table #5437: 536609562 bytes OK (needs compaction) 2024/11/19-15:28:15.858118 139850967680768 (Original Log Time 2024/11/19-15:28:15.835094) [memtable_list.cc:560] [default] Level-0 commit table #5437 started 2024/11/19-15:28:15.858121 139850967680768 (Original Log Time 2024/11/19-15:28:15.858038) [memtable_list.cc:759] [default] Level-0 commit table #5437: memtable #1 done 2024/11/19-15:28:15.858124 139850967680768 (Original Log Time 2024/11/19-15:28:15.858063) EVENT_LOG_v1 {"time_micros": 1732001295858057, "job": 269, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [3, 141, 61, 82, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:28:15.858127 139850967680768 (Original Log Time 2024/11/19-15:28:15.858083) [db_impl_compaction_flush.cc:310] [default] Level summary: files[3 141 61 82 0 0 0] max score 6.96 (177 files need compaction) 2024/11/19-15:28:15.979562 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-01 2024/11/19-15:28:15.979641 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00263/att-01/compact.done) = No such file or directory: 9.223 sec on http probe failed: 4, retry 2024/11/19-15:28:16.269740 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-01 2024/11/19-15:28:16.269826 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00265/att-01/compact.done) = No such file or directory: 5.204 sec on http probe failed: 2, retry 2024/11/19-15:28:16.371287 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-02 2024/11/19-15:28:16.371369 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00261/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:16.873431 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-01 2024/11/19-15:28:16.873530 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00266/att-01/compact.done) = No such file or directory: 5.202 sec on http probe failed: 2, retry 2024/11/19-15:28:17.255121 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-01 2024/11/19-15:28:17.255226 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00262/att-01/compact.done) = No such file or directory: 11.228 sec on http probe failed: 5, giveup 2024/11/19-15:28:17.255244 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00262/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.819 sec, real 11.514 (submit 278.339 ms wait 11.228 s), input 127.819 MiB, send shutdown 2024/11/19-15:28:17.260926 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-01 2024/11/19-15:28:17.261156 139851158521600 [dcompact_etcd.cc:1773] job-00262/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:17.266964 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-00 2024/11/19-15:28:17.267025 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00268/att-00/compact.done) = No such file or directory: 7.213 sec on http probe failed: 3, retry 2024/11/19-15:28:17.577184 139851158521600 [dcompact_etcd.cc:1163] job-00262/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.316 sec, input = 127.819 MiB 2024/11/19-15:28:17.857024 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-01 2024/11/19-15:28:17.857104 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00267/att-01/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:28:17.988483 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-01 2024/11/19-15:28:17.988564 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00263/att-01/compact.done) = No such file or directory: 11.228 sec on http probe failed: 5, giveup 2024/11/19-15:28:17.988582 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00263/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.818 sec, real 11.521 (submit 283.978 ms wait 11.228 s), input 127.818 MiB, send shutdown 2024/11/19-15:28:17.993441 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-01 2024/11/19-15:28:17.993656 139850978166528 [dcompact_etcd.cc:1773] job-00263/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:18.269786 139850978166528 [dcompact_etcd.cc:1163] job-00263/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.276 sec, input = 127.818 MiB 2024/11/19-15:28:18.275927 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-01 2024/11/19-15:28:18.275988 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00265/att-01/compact.done) = No such file or directory: 7.212 sec on http probe failed: 3, retry 2024/11/19-15:28:18.379391 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-02 2024/11/19-15:28:18.379482 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00261/att-02/compact.done) = No such file or directory: 5.207 sec on http probe failed: 2, retry 2024/11/19-15:28:18.882655 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-01 2024/11/19-15:28:18.882759 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00266/att-01/compact.done) = No such file or directory: 7.207 sec on http probe failed: 3, retry 2024/11/19-15:28:19.275684 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-00 2024/11/19-15:28:19.275791 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00268/att-00/compact.done) = No such file or directory: 9.222 sec on http probe failed: 4, retry 2024/11/19-15:28:19.863135 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-01 2024/11/19-15:28:19.863226 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00267/att-01/compact.done) = No such file or directory: 7.211 sec on http probe failed: 3, retry 2024/11/19-15:28:20.284594 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-01 2024/11/19-15:28:20.284674 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00265/att-01/compact.done) = No such file or directory: 9.219 sec on http probe failed: 4, retry 2024/11/19-15:28:20.387650 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-02 2024/11/19-15:28:20.387720 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00261/att-02/compact.done) = No such file or directory: 7.215 sec on http probe failed: 3, retry 2024/11/19-15:28:20.781245 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-02 2024/11/19-15:28:20.781330 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00262/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:20.892526 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-01 2024/11/19-15:28:20.892622 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00266/att-01/compact.done) = No such file or directory: 9.217 sec on http probe failed: 4, retry 2024/11/19-15:28:21.284868 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-00 2024/11/19-15:28:21.284968 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00268/att-00/compact.done) = No such file or directory: 11.230 sec on http probe failed: 5, giveup 2024/11/19-15:28:21.284988 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00268/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.823 sec, real 11.548 (submit 308.351 ms wait 11.230 s), input 127.823 MiB, send shutdown 2024/11/19-15:28:21.293900 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-00 2024/11/19-15:28:21.294142 139851169007360 [dcompact_etcd.cc:1773] job-00268/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:21.474926 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-02 2024/11/19-15:28:21.475013 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00263/att-02/compact.done) = No such file or directory: 3.195 sec on http probe failed: 1, retry 2024/11/19-15:28:21.558096 139851169007360 [dcompact_etcd.cc:1163] job-00268/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.264 sec, input = 127.823 MiB 2024/11/19-15:28:21.868406 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-01 2024/11/19-15:28:21.868481 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00267/att-01/compact.done) = No such file or directory: 9.217 sec on http probe failed: 4, retry 2024/11/19-15:28:22.289986 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-01 2024/11/19-15:28:22.290087 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00265/att-01/compact.done) = No such file or directory: 11.227 sec on http probe failed: 5, giveup 2024/11/19-15:28:22.290105 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00265/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.818 sec, real 11.513 (submit 280.084 ms wait 11.227 s), input 127.818 MiB, send shutdown 2024/11/19-15:28:22.298520 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-01 2024/11/19-15:28:22.298754 139851179493120 [dcompact_etcd.cc:1773] job-00265/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:22.396274 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-02 2024/11/19-15:28:22.396348 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00261/att-02/compact.done) = No such file or directory: 9.223 sec on http probe failed: 4, retry 2024/11/19-15:28:22.589716 139851179493120 [dcompact_etcd.cc:1163] job-00265/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.291 sec, input = 127.818 MiB 2024/11/19-15:28:22.787416 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-02 2024/11/19-15:28:22.787493 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00262/att-02/compact.done) = No such file or directory: 5.204 sec on http probe failed: 2, retry 2024/11/19-15:28:22.900484 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-01 2024/11/19-15:28:22.900579 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00266/att-01/compact.done) = No such file or directory: 11.226 sec on http probe failed: 5, giveup 2024/11/19-15:28:22.900597 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00266/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.522 (submit 287.689 ms wait 11.226 s), input 127.820 MiB, send shutdown 2024/11/19-15:28:22.909058 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-01 2024/11/19-15:28:22.909309 139851137550080 [dcompact_etcd.cc:1773] job-00266/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:23.192462 139851137550080 [dcompact_etcd.cc:1163] job-00266/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.283 sec, input = 127.820 MiB 2024/11/19-15:28:23.483864 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-02 2024/11/19-15:28:23.483952 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00263/att-02/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:28:23.874716 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-01 2024/11/19-15:28:23.874803 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00267/att-01/compact.done) = No such file or directory: 11.222 sec on http probe failed: 5, giveup 2024/11/19-15:28:23.874821 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00267/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.822 sec, real 11.499 (submit 270.661 ms wait 11.222 s), input 127.822 MiB, send shutdown 2024/11/19-15:28:23.883160 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-01 2024/11/19-15:28:23.883393 139851127064320 [dcompact_etcd.cc:1773] job-00267/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:24.163547 139851127064320 [dcompact_etcd.cc:1163] job-00267/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 127.822 MiB 2024/11/19-15:28:24.404723 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-02 2024/11/19-15:28:24.404800 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00261/att-02/compact.done) = No such file or directory: 11.232 sec on http probe failed: 5, giveup 2024/11/19-15:28:24.404819 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00261/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.819 sec, real 11.530 (submit 289.343 ms wait 11.232 s), input 127.819 MiB, send shutdown 2024/11/19-15:28:24.410385 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00261/att-02 2024/11/19-15:28:24.410594 139851148035840 [dcompact_etcd.cc:1773] job-00261/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:24.412450 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00261) = success 2024/11/19-15:28:24.412461 139851148035840 [dcompact_etcd.cc:1773] job-00261/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:28:24.412479 139851148035840 (Original Log Time 2024/11/19-15:27:49.807663) EVENT_LOG_v1 {"time_micros": 1732001269807655, "job": 261, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5274, 5280, 5286, 5293], "score": 8.70815, "input_data_size": 133300172, "oldest_snapshot_seqno": 153085840} 2024/11/19-15:28:24.412486 139851148035840 [compaction_job.cc:2437] [default] [JOB 261] Compacting 4@1 files to L2, score 8.71, subcompactions 1 : 1 2024/11/19-15:28:24.412494 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 259 Base level 1, inputs: [5274(31MB) 5280(31MB) 5286(31MB) 5293(31MB)] 2024/11/19-15:28:24.412712 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005438.sst) /storage/mytopling/..//tmp/Topling-82038-qN99kv 2024/11/19-15:28:24.556151 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005438.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:28:24.590598 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005438.sst): index pass time = 0.03's, 215.448'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463359 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:28:24.656874 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005438.sst): first pass time = 0.24's, 274.870'MB/sec 2024/11/19-15:28:24.656942 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005438.sst): old prealloc_size = 73819750, real_size = 55606304 2024/11/19-15:28:24.657170 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005438.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463359 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.83 avg-zval = 117.86 2024/11/19-15:28:24.657225 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005438.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.101 sec, Key+Value = 0.067 GB 2024/11/19-15:28:24.750101 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005438.sst): second pass time = 0.00's,196854334.545'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 595.745'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3901.004'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.663'MB/sec write SST data time = 0.05's, 1122.596'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 196854334.545'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463359 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.83 avg-zval = 117.86 usrkeys = 463359 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.83 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6616 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5724 value = 0.9912 dict = 0.00 all = 1.2093 } Zip/UnZip{ index = 0.0946 value = 1.0088 dict = 0.00 all = 0.8269 } ---------------------------- total value len = 19.159193 GB avg = 0.274 KB (by entry num) total key len = 1.791229 GB avg = 0.026 KB total ukey len = 1.231889 GB avg = 0.018 KB total ukey num = 0.069917468 Billion total entry num = 0.069917468 Billion write speed all = 35.643188579 MB/sec (with seq num) write speed all = 34.691577584 MB/sec (without seq num) 2024/11/19-15:28:24.753671 139851148035840 [compaction_job.cc:1948] [default] [JOB 261] Generated table #5438: 463359 keys, 55495768 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:24.753723 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001304753700, "cf_name": "default", "job": 261, "event": "table_file_creation", "file_number": 5438, "file_size": 55495768, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54613712, "index_size": 876544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12974052, "raw_average_key_size": 28, "raw_value_size": 54134942, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463359, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001304, "oldest_key_time": 0, "file_creation_time": 1732001304, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5438, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:24.759498 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-01 2024/11/19-15:28:24.759593 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00268/att-01/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:24.767896 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005439.sst) /storage/mytopling/..//tmp/Topling-82038-rFrQ5d 2024/11/19-15:28:24.792215 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-02 2024/11/19-15:28:24.792293 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00262/att-02/compact.done) = No such file or directory: 7.210 sec on http probe failed: 3, retry 2024/11/19-15:28:24.911384 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005439.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:28:24.945709 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005439.sst): index pass time = 0.03's, 215.661'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462179 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:28:25.011831 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005439.sst): first pass time = 0.24's, 274.350'MB/sec 2024/11/19-15:28:25.011883 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005439.sst): old prealloc_size = 73819750, real_size = 55465568 2024/11/19-15:28:25.012117 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005439.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462179 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.79 avg-zval = 117.85 2024/11/19-15:28:25.012181 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005439.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.101 sec, Key+Value = 0.067 GB 2024/11/19-15:28:25.108884 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005439.sst): second pass time = 0.00's,107098250.000'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 570.750'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3789.019'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 2.432'MB/sec write SST data time = 0.05's, 1123.886'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 107098250.000'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462179 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 117.85 usrkeys = 462179 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6597 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0554 GB } UnZip/Zip{ index = 10.5033 value = 0.9910 dict = 0.00 all = 1.2089 } Zip/UnZip{ index = 0.0952 value = 1.0091 dict = 0.00 all = 0.8272 } ---------------------------- total value len = 19.213171 GB avg = 0.273 KB (by entry num) total key len = 1.804170 GB avg = 0.026 KB total ukey len = 1.241133 GB avg = 0.018 KB total ukey num = 0.070379647 Billion total entry num = 0.070379647 Billion write speed all = 35.735224802 MB/sec (with seq num) write speed all = 34.777907678 MB/sec (without seq num) 2024/11/19-15:28:25.112373 139851148035840 [compaction_job.cc:1948] [default] [JOB 261] Generated table #5439: 462179 keys, 55355352 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:25.112423 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001305112401, "cf_name": "default", "job": 261, "event": "table_file_creation", "file_number": 5439, "file_size": 55355352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54469776, "index_size": 880064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12941012, "raw_average_key_size": 28, "raw_value_size": 53977518, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462179, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001304, "oldest_key_time": 0, "file_creation_time": 1732001304, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5439, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:25.125751 139851148035840 [compaction_job.cc:703] job-00261: subcompact[0], size: 0.110851 G, files: /storage/mytopling/.rocksdb [005438,005439] 2024/11/19-15:28:25.127212 139851148035840 (Original Log Time 2024/11/19-15:28:24.412520) EVENT_LOG_v1 {"time_micros": 1732001304412505, "job": 261, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5274, 5280, 5286, 5293], "score": 8.70815, "input_data_size": 133300172, "oldest_snapshot_seqno": 153085840} 2024/11/19-15:28:25.127222 139851148035840 (Original Log Time 2024/11/19-15:28:25.126631) [compaction_job.cc:2022] [default] [JOB 261] Compacted 4@1 files to L2 => 110851120 bytes 2024/11/19-15:28:25.127226 139851148035840 (Original Log Time 2024/11/19-15:28:25.127068) [compaction_job.cc:1224] [default] compacted to: files[3 137 63 82 0 0 0] max score 6.96 (179 files need compaction), MB/sec: 186.9 rd, 155.4 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925538, records dropped: 0 output_compression: Snappy 2024/11/19-15:28:25.127245 139851148035840 (Original Log Time 2024/11/19-15:28:25.127094) EVENT_LOG_v1 {"time_micros": 1732001305127082, "cf": "default", "job": 261, "event": "compaction_finished", "compaction_time_micros": 713246, "compaction_time_cpu_micros": 710365, "output_level": 2, "num_output_files": 2, "total_output_size": 110851120, "num_input_records": 925538, "num_output_records": 925538, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 137, 63, 82, 0, 0, 0]} 2024/11/19-15:28:25.129948 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004981.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:25.129973 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001305129970, "job": 261, "event": "table_file_deletion", "file_number": 4981} 2024/11/19-15:28:25.137152 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004982.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:25.137174 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001305137172, "job": 261, "event": "table_file_deletion", "file_number": 4982} 2024/11/19-15:28:25.145214 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004983.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:25.145235 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001305145233, "job": 261, "event": "table_file_deletion", "file_number": 4983} 2024/11/19-15:28:25.153899 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004986.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:25.153996 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001305153917, "job": 261, "event": "table_file_deletion", "file_number": 4986} 2024/11/19-15:28:25.154147 139851148035840 [compaction_job.cc:2437] [default] [JOB 270] Compacting 1@1 files to L2, score 6.96, subcompactions 1 : 1 2024/11/19-15:28:25.154156 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 268 Base level 1, inputs: [5377(24MB)] 2024/11/19-15:28:25.446586 139851148035840 [dcompact_etcd.cc:1163] job-00270/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.292 sec, input = 24.862 MiB 2024/11/19-15:28:25.491967 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-02 2024/11/19-15:28:25.492026 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00263/att-02/compact.done) = No such file or directory: 7.214 sec on http probe failed: 3, retry 2024/11/19-15:28:25.794863 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-02 2024/11/19-15:28:25.794956 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00265/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:26.077742 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-00 2024/11/19-15:28:26.077834 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00270/att-00/compact.done) = No such file or directory: 0.622 sec on http probe failed: 1, retry 2024/11/19-15:28:26.397760 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-02 2024/11/19-15:28:26.397851 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00266/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:26.466072 139852349699840 [db_impl_write.cc:1661] Flushing all column families with data in WAL number 4798. Total log size is 8726663754 while max_total_wal_size is 8724152320 2024/11/19-15:28:26.466174 139852349699840 [db_impl_write.cc:2181] [__system__] New memtable created with log file: #5440. Immutable memtables: 0. 2024/11/19-15:28:26.466585 139852349699840 [cspp_memtable.cc:1074] ConvertToReadOnly in MarkReadOnly, mem_size = 0.375 M, time = 8.557 us 2024/11/19-15:28:26.466681 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 271] Syncing log #5436 2024/11/19-15:28:26.466776 139850957195008 (Original Log Time 2024/11/19-15:28:26.466659) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [__system__], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:28:26.466783 139850957195008 [flush_job.cc:879] [__system__] [JOB 271] Flushing memtable with next log file: 5440 2024/11/19-15:28:26.466804 139850957195008 EVENT_LOG_v1 {"time_micros": 1732001306466798, "job": 271, "event": "flush_started", "num_memtables": 1, "num_entries": 4136, "num_deletes": 0, "total_data_size": 330579, "memory_usage": 393608, "num_range_deletes": 0, "flush_reason": "WAL Full"} 2024/11/19-15:28:26.466807 139850957195008 [flush_job.cc:910] [__system__] [JOB 271] Level-0 flush table #5441: started 2024/11/19-15:28:26.467310 139850957195008 EVENT_LOG_v1 {"time_micros": 1732001306467290, "cf_name": "__system__", "job": 271, "event": "table_file_creation", "file_number": 5441, "file_size": 4442, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 133973428, "largest_seqno": 155940692, "table_properties": {"data_size": 960, "index_size": 2688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 192, "raw_average_key_size": 19, "raw_value_size": 909, "raw_average_value_size": 90, "num_data_blocks": 1, "num_entries": 10, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "__system__", "column_family_id": 1, "comparator": "leveldb.BytewiseComparator", "merge_operator": "Rdb_system_merge_op", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "SngFast", "compression_options": "", "creation_time": 1732001306, "oldest_key_time": 1732000876, "file_creation_time": 1732001306, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5441, "seqno_to_time_mapping": "N/A"}} 2024/11/19-15:28:26.467798 139850957195008 [flush_job.cc:1083] [__system__] [JOB 271] Flush lasted 1023 microseconds, and 1000 cpu microseconds. 2024/11/19-15:28:26.467934 139850957195008 (Original Log Time 2024/11/19-15:28:26.467323) [flush_job.cc:1029] [__system__] [JOB 271] Level-0 flush table #5441: 4442 bytes OK (needs compaction) 2024/11/19-15:28:26.467940 139850957195008 (Original Log Time 2024/11/19-15:28:26.467807) [memtable_list.cc:560] [__system__] Level-0 commit table #5441 started 2024/11/19-15:28:26.467943 139850957195008 (Original Log Time 2024/11/19-15:28:26.467859) [memtable_list.cc:759] [__system__] Level-0 commit table #5441: memtable #1 done 2024/11/19-15:28:26.467946 139850957195008 (Original Log Time 2024/11/19-15:28:26.467873) EVENT_LOG_v1 {"time_micros": 1732001306467867, "job": 271, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:28:26.467949 139850957195008 (Original Log Time 2024/11/19-15:28:26.467897) [db_impl_compaction_flush.cc:310] [__system__] Level summary: files[1 1 0 0 0 0 0] max score 0.25 (1 files need compaction) 2024/11/19-15:28:26.467962 139850957195008 [db_impl_files.cc:470] [JOB 271] Try to delete WAL files size 7926994849, prev total WAL file size 8726663875, number of live WAL files 21. 2024/11/19-15:28:26.477939 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004798.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.481220 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004812.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.493828 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004903.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.504821 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004942.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.517335 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004960.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.529455 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004965.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.540169 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004976.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.558653 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004984.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.571765 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005003.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.583876 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-00 2024/11/19-15:28:26.583973 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00270/att-00/compact.done) = No such file or directory: 1.129 sec on http probe failed: 2, retry 2024/11/19-15:28:26.588854 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005058.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.626979 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005167.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.683099 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005172.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.733997 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005187.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.765509 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-01 2024/11/19-15:28:26.765597 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00268/att-01/compact.done) = No such file or directory: 5.202 sec on http probe failed: 2, retry 2024/11/19-15:28:26.782206 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005191.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.801826 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-02 2024/11/19-15:28:26.801891 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00262/att-02/compact.done) = No such file or directory: 9.215 sec on http probe failed: 4, retry 2024/11/19-15:28:26.834338 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005218.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.889549 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005231.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.940661 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005246.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:26.995960 139850957195008 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005417.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:27.090589 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-00 2024/11/19-15:28:27.090682 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00270/att-00/compact.done) = No such file or directory: 1.635 sec on http probe failed: 3, retry 2024/11/19-15:28:27.369326 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-02 2024/11/19-15:28:27.369410 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00267/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:27.501153 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-02 2024/11/19-15:28:27.501232 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00263/att-02/compact.done) = No such file or directory: 9.222 sec on http probe failed: 4, retry 2024/11/19-15:28:27.597488 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-00 2024/11/19-15:28:27.597557 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00270/att-00/compact.done) = No such file or directory: 2.141 sec on http probe failed: 4, retry 2024/11/19-15:28:27.804470 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-02 2024/11/19-15:28:27.804547 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00265/att-02/compact.done) = No such file or directory: 5.205 sec on http probe failed: 2, retry 2024/11/19-15:28:28.103966 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-00 2024/11/19-15:28:28.104041 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00270/att-00/compact.done) = No such file or directory: 2.648 sec on http probe failed: 5, giveup 2024/11/19-15:28:28.104059 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00270/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 24.862 sec, real 2.949 (submit 292.009 ms wait 2.648 s), input 24.862 MiB, send shutdown 2024/11/19-15:28:28.112601 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-00 2024/11/19-15:28:28.112813 139851148035840 [dcompact_etcd.cc:1773] job-00270/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:28.410777 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-02 2024/11/19-15:28:28.410874 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00266/att-02/compact.done) = No such file or directory: 5.206 sec on http probe failed: 2, retry 2024/11/19-15:28:28.418340 139851148035840 [dcompact_etcd.cc:1163] job-00270/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.305 sec, input = 24.862 MiB 2024/11/19-15:28:28.771679 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-01 2024/11/19-15:28:28.771775 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00268/att-01/compact.done) = No such file or directory: 7.208 sec on http probe failed: 3, retry 2024/11/19-15:28:28.810628 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-02 2024/11/19-15:28:28.810701 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00262/att-02/compact.done) = No such file or directory: 11.225 sec on http probe failed: 5, giveup 2024/11/19-15:28:28.810720 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00262/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.819 sec, real 11.549 (submit 315.945 ms wait 11.225 s), input 127.819 MiB, send shutdown 2024/11/19-15:28:28.818832 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00262/att-02 2024/11/19-15:28:28.819060 139851158521600 [dcompact_etcd.cc:1773] job-00262/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:28.820864 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00262) = success 2024/11/19-15:28:28.820872 139851158521600 [dcompact_etcd.cc:1773] job-00262/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:28:28.820891 139851158521600 (Original Log Time 2024/11/19-15:27:54.202579) EVENT_LOG_v1 {"time_micros": 1732001274202571, "job": 262, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5297, 5300, 5303, 5305], "score": 8.45851, "input_data_size": 133300300, "oldest_snapshot_seqno": -1} 2024/11/19-15:28:28.820899 139851158521600 [compaction_job.cc:2437] [default] [JOB 262] Compacting 4@1 files to L2, score 8.46, subcompactions 1 : 1 2024/11/19-15:28:28.820907 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 260 Base level 1, inputs: [5297(31MB) 5300(31MB) 5303(31MB) 5305(31MB)] 2024/11/19-15:28:28.821137 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005442.sst) /storage/mytopling/..//tmp/Topling-82038-1Wq7aW 2024/11/19-15:28:28.963733 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005442.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:28:28.997867 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005442.sst): index pass time = 0.03's, 217.468'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463495 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:28:29.049374 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-01 2024/11/19-15:28:29.049478 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00270/att-01/compact.done) = No such file or directory: 0.622 sec on http probe failed: 1, retry 2024/11/19-15:28:29.082503 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005442.sst): first pass time = 0.26's, 256.776'MB/sec 2024/11/19-15:28:29.082570 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005442.sst): old prealloc_size = 73819750, real_size = 55617552 2024/11/19-15:28:29.082809 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005442.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463495 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.79 avg-zval = 117.85 2024/11/19-15:28:29.082868 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005442.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.119 sec, Key+Value = 0.067 GB 2024/11/19-15:28:29.186345 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005442.sst): second pass time = 0.00's,196127253.623'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 534.875'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3760.717'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 2.463'MB/sec write SST data time = 0.06's, 982.467'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 196127253.623'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463495 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.79 avg-zval = 117.85 usrkeys = 463495 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.79 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6614 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5740 value = 0.9910 dict = 0.00 all = 1.2090 } Zip/UnZip{ index = 0.0946 value = 1.0091 dict = 0.00 all = 0.8271 } ---------------------------- total value len = 19.267302 GB avg = 0.272 KB (by entry num) total key len = 1.817148 GB avg = 0.026 KB total ukey len = 1.250403 GB avg = 0.018 KB total ukey num = 0.070843142 Billion total entry num = 0.070843142 Billion write speed all = 35.602503530 MB/sec (with seq num) write speed all = 34.645516456 MB/sec (without seq num) 2024/11/19-15:28:29.190585 139851158521600 [compaction_job.cc:1948] [default] [JOB 262] Generated table #5442: 463495 keys, 55506968 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:29.190639 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001309190615, "cf_name": "default", "job": 262, "event": "table_file_creation", "file_number": 5442, "file_size": 55506968, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54624832, "index_size": 876672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12977860, "raw_average_key_size": 28, "raw_value_size": 54131122, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463495, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001308, "oldest_key_time": 0, "file_creation_time": 1732001308, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5442, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:29.206125 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005443.sst) /storage/mytopling/..//tmp/Topling-82038-g7mH9n 2024/11/19-15:28:29.346950 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005443.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:28:29.378742 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-02 2024/11/19-15:28:29.378848 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00267/att-02/compact.done) = No such file or directory: 5.206 sec on http probe failed: 2, retry 2024/11/19-15:28:29.381211 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005443.sst): index pass time = 0.03's, 216.012'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462050 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:28:29.453537 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005443.sst): first pass time = 0.25's, 270.495'MB/sec 2024/11/19-15:28:29.453591 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005443.sst): old prealloc_size = 73819750, real_size = 55455152 2024/11/19-15:28:29.453819 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005443.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462050 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.83 avg-zval = 117.86 2024/11/19-15:28:29.453878 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005443.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.107 sec, Key+Value = 0.067 GB 2024/11/19-15:28:29.510247 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-02 2024/11/19-15:28:29.510367 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00263/att-02/compact.done) = No such file or directory: 11.232 sec on http probe failed: 5, giveup 2024/11/19-15:28:29.510392 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00263/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.818 sec, real 11.517 (submit 276.050 ms wait 11.232 s), input 127.818 MiB, send shutdown 2024/11/19-15:28:29.515264 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00263/att-02 2024/11/19-15:28:29.529197 139850978166528 [dcompact_etcd.cc:1773] job-00263/att-02: CleanFiles 0.014 sec : 0.000 2024/11/19-15:28:29.549929 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00263) = success 2024/11/19-15:28:29.549947 139850978166528 [dcompact_etcd.cc:1773] job-00263/att-03: CleanFiles 0.021 sec : 0.000 2024/11/19-15:28:29.549973 139850978166528 (Original Log Time 2024/11/19-15:27:54.952149) EVENT_LOG_v1 {"time_micros": 1732001274952142, "job": 263, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5307, 5312, 5316, 5318], "score": 8.20886, "input_data_size": 133300300, "oldest_snapshot_seqno": -1} 2024/11/19-15:28:29.549983 139850978166528 [compaction_job.cc:2437] [default] [JOB 263] Compacting 4@1 files to L2, score 8.21, subcompactions 1 : 1 2024/11/19-15:28:29.549995 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 261 Base level 1, inputs: [5307(31MB) 5312(31MB) 5316(31MB) 5318(31MB)] 2024/11/19-15:28:29.550277 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005444.sst) /storage/mytopling/..//tmp/Topling-82038-jC7bqm 2024/11/19-15:28:29.551812 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-01 2024/11/19-15:28:29.551870 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00270/att-01/compact.done) = No such file or directory: 1.128 sec on http probe failed: 2, retry 2024/11/19-15:28:29.558661 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005443.sst): second pass time = 0.00's,190077098.592'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.11's, 526.896'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3952.551'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 2.375'MB/sec write SST data time = 0.06's, 984.992'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 190077098.592'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462050 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.83 avg-zval = 117.86 usrkeys = 462050 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.83 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6597 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0553 GB } UnZip/Zip{ index = 10.5011 value = 0.9912 dict = 0.00 all = 1.2091 } Zip/UnZip{ index = 0.0952 value = 1.0088 dict = 0.00 all = 0.8270 } ---------------------------- total value len = 19.321284 GB avg = 0.271 KB (by entry num) total key len = 1.830085 GB avg = 0.026 KB total ukey len = 1.259644 GB avg = 0.018 KB total ukey num = 0.071305192 Billion total entry num = 0.071305192 Billion write speed all = 35.693062402 MB/sec (with seq num) write speed all = 34.730438873 MB/sec (without seq num) 2024/11/19-15:28:29.577865 139851158521600 [compaction_job.cc:1948] [default] [JOB 262] Generated table #5443: 462050 keys, 55344920 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:29.577926 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001309577895, "cf_name": "default", "job": 262, "event": "table_file_creation", "file_number": 5443, "file_size": 55344920, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54459424, "index_size": 880000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12937400, "raw_average_key_size": 28, "raw_value_size": 53981896, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462050, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001309, "oldest_key_time": 0, "file_creation_time": 1732001309, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5443, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:29.593602 139851158521600 [compaction_job.cc:703] job-00262: subcompact[0], size: 0.110852 G, files: /storage/mytopling/.rocksdb [005442,005443] 2024/11/19-15:28:29.595088 139851158521600 (Original Log Time 2024/11/19-15:28:28.820945) EVENT_LOG_v1 {"time_micros": 1732001308820928, "job": 262, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5297, 5300, 5303, 5305], "score": 8.45851, "input_data_size": 133300300, "oldest_snapshot_seqno": -1} 2024/11/19-15:28:29.595098 139851158521600 (Original Log Time 2024/11/19-15:28:29.594593) [compaction_job.cc:2022] [default] [JOB 262] Compacted 4@1 files to L2 => 110851888 bytes 2024/11/19-15:28:29.595102 139851158521600 (Original Log Time 2024/11/19-15:28:29.594964) [compaction_job.cc:1224] [default] compacted to: files[3 133 65 82 0 0 0] max score 6.91 (180 files need compaction), MB/sec: 172.5 rd, 143.5 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925545, records dropped: 0 output_compression: Snappy 2024/11/19-15:28:29.595122 139851158521600 (Original Log Time 2024/11/19-15:28:29.594992) EVENT_LOG_v1 {"time_micros": 1732001309594979, "cf": "default", "job": 262, "event": "compaction_finished", "compaction_time_micros": 772675, "compaction_time_cpu_micros": 752139, "output_level": 2, "num_output_files": 2, "total_output_size": 110851888, "num_input_records": 925545, "num_output_records": 925545, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 133, 65, 82, 0, 0, 0]} 2024/11/19-15:28:29.598903 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005266.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:29.598934 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001309598931, "job": 262, "event": "table_file_deletion", "file_number": 5266} 2024/11/19-15:28:29.599056 139851158521600 [compaction_job.cc:2437] [default] [JOB 272] Compacting 4@1 files to L2, score 6.91, subcompactions 1 : 1 2024/11/19-15:28:29.599064 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 270 Base level 1, inputs: [5380(31MB) 5381(24MB) 5252(31MB) 5255(31MB)] 2024/11/19-15:28:29.709758 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005444.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:28:29.761256 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005444.sst): index pass time = 0.05's, 144.151'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463601 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:28:29.812723 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-02 2024/11/19-15:28:29.812821 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00265/att-02/compact.done) = No such file or directory: 7.215 sec on http probe failed: 3, retry 2024/11/19-15:28:29.815673 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005444.sst): first pass time = 0.27's, 252.881'MB/sec 2024/11/19-15:28:29.815721 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005444.sst): old prealloc_size = 73819750, real_size = 55632608 2024/11/19-15:28:29.815963 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005444.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463601 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.76 avg-zval = 117.85 2024/11/19-15:28:29.816017 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005444.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.106 sec, Key+Value = 0.067 GB 2024/11/19-15:28:29.877504 139851158521600 [dcompact_etcd.cc:1163] job-00272/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.278 sec, input = 119.253 MiB 2024/11/19-15:28:29.911736 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005444.sst): second pass time = 0.00's,189259335.664'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 578.340'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3698.261'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 2.555'MB/sec write SST data time = 0.05's, 1101.578'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 189259335.664'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463601 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 117.85 usrkeys = 463601 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6615 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5005 value = 0.9908 dict = 0.00 all = 1.2087 } Zip/UnZip{ index = 0.0952 value = 1.0093 dict = 0.00 all = 0.8273 } ---------------------------- total value len = 19.375412 GB avg = 0.270 KB (by entry num) total key len = 1.843066 GB avg = 0.026 KB total ukey len = 1.268916 GB avg = 0.018 KB total ukey num = 0.071768793 Billion total entry num = 0.071768793 Billion write speed all = 35.784987014 MB/sec (with seq num) write speed all = 34.816681807 MB/sec (without seq num) 2024/11/19-15:28:29.915646 139850978166528 [compaction_job.cc:1948] [default] [JOB 263] Generated table #5444: 463601 keys, 55522008 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:29.915695 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001309915673, "cf_name": "default", "job": 263, "event": "table_file_creation", "file_number": 5444, "file_size": 55522008, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54633488, "index_size": 883008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12980828, "raw_average_key_size": 28, "raw_value_size": 54128170, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463601, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001309, "oldest_key_time": 0, "file_creation_time": 1732001309, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5444, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:29.930635 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005445.sst) /storage/mytopling/..//tmp/Topling-82038-9fCt6w 2024/11/19-15:28:30.057804 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-01 2024/11/19-15:28:30.057879 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00270/att-01/compact.done) = No such file or directory: 1.631 sec on http probe failed: 3, retry 2024/11/19-15:28:30.072646 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005445.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:28:30.118485 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005445.sst): index pass time = 0.05's, 161.384'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462019 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:28:30.196386 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005445.sst): first pass time = 0.27's, 251.822'MB/sec 2024/11/19-15:28:30.196441 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005445.sst): old prealloc_size = 73819750, real_size = 55446288 2024/11/19-15:28:30.196698 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005445.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462019 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.84 avg-zval = 117.87 2024/11/19-15:28:30.196767 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005445.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.124 sec, Key+Value = 0.067 GB 2024/11/19-15:28:30.293083 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005445.sst): second pass time = 0.00's,160181050.445'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 572.678'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3470.206'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 2.540'MB/sec write SST data time = 0.05's, 1087.157'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 160181050.445'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462019 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.84 avg-zval = 117.87 usrkeys = 462019 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.84 avg-zval = 117.87 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6596 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0553 GB } UnZip/Zip{ index = 10.5704 value = 0.9913 dict = 0.00 all = 1.2093 } Zip/UnZip{ index = 0.0946 value = 1.0088 dict = 0.00 all = 0.8269 } ---------------------------- total value len = 19.429393 GB avg = 0.269 KB (by entry num) total key len = 1.856002 GB avg = 0.026 KB total ukey len = 1.278156 GB avg = 0.018 KB total ukey num = 0.072230812 Billion total entry num = 0.072230812 Billion write speed all = 35.874771112 MB/sec (with seq num) write speed all = 34.900858700 MB/sec (without seq num) 2024/11/19-15:28:30.297051 139850978166528 [compaction_job.cc:1948] [default] [JOB 263] Generated table #5445: 462019 keys, 55336088 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:30.297102 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001310297079, "cf_name": "default", "job": 263, "event": "table_file_creation", "file_number": 5445, "file_size": 55336088, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54456448, "index_size": 874176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12936532, "raw_average_key_size": 28, "raw_value_size": 53981014, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462019, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001309, "oldest_key_time": 0, "file_creation_time": 1732001309, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5445, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:30.311937 139850978166528 [compaction_job.cc:703] job-00263: subcompact[0], size: 0.110858 G, files: /storage/mytopling/.rocksdb [005444,005445] 2024/11/19-15:28:30.313318 139850978166528 (Original Log Time 2024/11/19-15:28:29.550032) EVENT_LOG_v1 {"time_micros": 1732001309550011, "job": 263, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5307, 5312, 5316, 5318], "score": 8.20886, "input_data_size": 133300300, "oldest_snapshot_seqno": -1} 2024/11/19-15:28:30.313327 139850978166528 (Original Log Time 2024/11/19-15:28:30.312834) [compaction_job.cc:2022] [default] [JOB 263] Compacted 4@1 files to L2 => 110858096 bytes 2024/11/19-15:28:30.313331 139850978166528 (Original Log Time 2024/11/19-15:28:30.313202) [compaction_job.cc:1224] [default] compacted to: files[3 129 67 82 0 0 0] max score 6.68 (178 files need compaction), MB/sec: 175.0 rd, 145.5 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925620, records dropped: 0 output_compression: Snappy 2024/11/19-15:28:30.313351 139850978166528 (Original Log Time 2024/11/19-15:28:30.313225) EVENT_LOG_v1 {"time_micros": 1732001310313214, "cf": "default", "job": 263, "event": "compaction_finished", "compaction_time_micros": 761918, "compaction_time_cpu_micros": 754720, "output_level": 2, "num_output_files": 2, "total_output_size": 110858096, "num_input_records": 925620, "num_output_records": 925620, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 129, 67, 82, 0, 0, 0]} 2024/11/19-15:28:30.318287 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005256.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:30.318312 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001310318309, "job": 263, "event": "table_file_deletion", "file_number": 5256} 2024/11/19-15:28:30.322552 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005258.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:30.322574 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001310322572, "job": 263, "event": "table_file_deletion", "file_number": 5258} 2024/11/19-15:28:30.327424 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005261.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:30.327444 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001310327442, "job": 263, "event": "table_file_deletion", "file_number": 5261} 2024/11/19-15:28:30.332029 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005264.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:30.332049 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001310332047, "job": 263, "event": "table_file_deletion", "file_number": 5264} 2024/11/19-15:28:30.332177 139850978166528 [compaction_job.cc:2437] [__system__] [JOB 273] Compacting 1@0 + 1@1 files to L1, score 0.25, subcompactions 3 : 1 2024/11/19-15:28:30.332185 139850978166528 [compaction_job.cc:2445] [__system__]: Compaction start summary: Base version 269 Base level 0, inputs: [5441(4442B)], [4936(5800B)] 2024/11/19-15:28:30.332485 139850978166528 [compaction_job.cc:1948] [__system__] [JOB 273] Generated table #5446: 34 keys, 5800 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:30.332515 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001310332496, "cf_name": "__system__", "job": 273, "event": "table_file_creation", "file_number": 5446, "file_size": 5800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 1344, "index_size": 3648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 803, "raw_average_key_size": 23, "raw_value_size": 1281, "raw_average_value_size": 37, "num_data_blocks": 1, "num_entries": 34, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "__system__", "column_family_id": 1, "comparator": "leveldb.BytewiseComparator", "merge_operator": "Rdb_system_merge_op", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "SngFast", "compression_options": "allseq0", "creation_time": 1732001310, "oldest_key_time": 0, "file_creation_time": 1732001310, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5446, "seqno_to_time_mapping": "N/A"}} 2024/11/19-15:28:30.332556 139850978166528 [compaction_job.cc:703] job-00273: subcompact[0], size: 0.000006 G, files: /storage/mytopling/.rocksdb [005446] 2024/11/19-15:28:30.332796 139850978166528 (Original Log Time 2024/11/19-15:28:30.332198) EVENT_LOG_v1 {"time_micros": 1732001310332191, "job": 273, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L0": [5441], "files_L1": [4936], "score": 0.25, "input_data_size": 10242, "oldest_snapshot_seqno": 156281970} 2024/11/19-15:28:30.332802 139850978166528 (Original Log Time 2024/11/19-15:28:30.332693) [compaction_job.cc:2022] [__system__] [JOB 273] Compacted 1@0 + 1@1 files to L1 => 5800 bytes 2024/11/19-15:28:30.332805 139850978166528 (Original Log Time 2024/11/19-15:28:30.332734) [compaction_job.cc:1224] [__system__] compacted to: files[0 1 0 0 0 0 0] max score 0.00, MB/sec: 28.2 rd, 16.0 wr, level 1, files in(1, 1) out(1 +0 blob) MB in(0.0, 0.0 +0.0 blob) out(0.0 +0.0 blob), read-write-amplify(3.6) write-amplify(1.3) OK, records in: 44, records dropped: 10 output_compression: Snappy 2024/11/19-15:28:30.332813 139850978166528 (Original Log Time 2024/11/19-15:28:30.332750) EVENT_LOG_v1 {"time_micros": 1732001310332742, "cf": "__system__", "job": 273, "event": "compaction_finished", "compaction_time_micros": 363, "compaction_time_cpu_micros": 268, "output_level": 1, "num_output_files": 1, "total_output_size": 5800, "num_input_records": 44, "num_output_records": 34, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]} 2024/11/19-15:28:30.332954 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/004936.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:30.332966 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001310332964, "job": 273, "event": "table_file_deletion", "file_number": 4936} 2024/11/19-15:28:30.333071 139850978166528 [compaction_job.cc:2437] [default] [JOB 274] Compacting 4@1 files to L2, score 6.68, subcompactions 1 : 1 2024/11/19-15:28:30.333079 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 271 Base level 1, inputs: [5257(31MB) 5259(31MB) 5262(31MB) 5263(31MB)] 2024/11/19-15:28:30.420965 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-02 2024/11/19-15:28:30.421065 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00266/att-02/compact.done) = No such file or directory: 7.218 sec on http probe failed: 3, retry 2024/11/19-15:28:30.563563 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-01 2024/11/19-15:28:30.563659 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00270/att-01/compact.done) = No such file or directory: 2.137 sec on http probe failed: 4, retry 2024/11/19-15:28:30.638309 139850978166528 [dcompact_etcd.cc:1163] job-00274/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.305 sec, input = 126.907 MiB 2024/11/19-15:28:30.776737 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-01 2024/11/19-15:28:30.776807 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00268/att-01/compact.done) = No such file or directory: 9.214 sec on http probe failed: 4, retry 2024/11/19-15:28:31.065567 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-01 2024/11/19-15:28:31.065646 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00270/att-01/compact.done) = No such file or directory: 2.643 sec on http probe failed: 5, giveup 2024/11/19-15:28:31.065665 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00270/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 24.862 sec, real 2.953 (submit 305.458 ms wait 2.643 s), input 24.862 MiB, send shutdown 2024/11/19-15:28:31.070686 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-01 2024/11/19-15:28:31.070908 139851148035840 [dcompact_etcd.cc:1773] job-00270/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:31.361977 139851148035840 [dcompact_etcd.cc:1163] job-00270/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.291 sec, input = 24.862 MiB 2024/11/19-15:28:31.386822 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-02 2024/11/19-15:28:31.386883 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00267/att-02/compact.done) = No such file or directory: 7.215 sec on http probe failed: 3, retry 2024/11/19-15:28:31.819119 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-02 2024/11/19-15:28:31.819225 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00265/att-02/compact.done) = No such file or directory: 9.223 sec on http probe failed: 4, retry 2024/11/19-15:28:31.992979 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-02 2024/11/19-15:28:31.993063 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00270/att-02/compact.done) = No such file or directory: 0.622 sec on http probe failed: 1, retry 2024/11/19-15:28:32.426067 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-02 2024/11/19-15:28:32.426161 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00266/att-02/compact.done) = No such file or directory: 9.229 sec on http probe failed: 4, retry 2024/11/19-15:28:32.498647 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-02 2024/11/19-15:28:32.498712 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00270/att-02/compact.done) = No such file or directory: 1.128 sec on http probe failed: 2, retry 2024/11/19-15:28:32.786989 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-01 2024/11/19-15:28:32.787064 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00268/att-01/compact.done) = No such file or directory: 11.219 sec on http probe failed: 5, giveup 2024/11/19-15:28:32.787083 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00268/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.823 sec, real 11.493 (submit 263.881 ms wait 11.219 s), input 127.823 MiB, send shutdown 2024/11/19-15:28:32.794970 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-01 2024/11/19-15:28:32.795183 139851169007360 [dcompact_etcd.cc:1773] job-00268/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:32.869082 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-00 2024/11/19-15:28:32.869170 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00272/att-00/compact.done) = No such file or directory: 2.981 sec on http probe failed: 1, retry 2024/11/19-15:28:33.005509 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-02 2024/11/19-15:28:33.005586 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00270/att-02/compact.done) = No such file or directory: 1.634 sec on http probe failed: 3, retry 2024/11/19-15:28:33.086250 139851169007360 [dcompact_etcd.cc:1163] job-00268/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.291 sec, input = 127.823 MiB 2024/11/19-15:28:33.392242 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-02 2024/11/19-15:28:33.392322 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00267/att-02/compact.done) = No such file or directory: 9.223 sec on http probe failed: 4, retry 2024/11/19-15:28:33.511042 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-02 2024/11/19-15:28:33.511125 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00270/att-02/compact.done) = No such file or directory: 2.141 sec on http probe failed: 4, retry 2024/11/19-15:28:33.816299 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-00 2024/11/19-15:28:33.816378 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00274/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:28:33.828221 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-02 2024/11/19-15:28:33.828281 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00265/att-02/compact.done) = No such file or directory: 11.230 sec on http probe failed: 5, giveup 2024/11/19-15:28:33.828297 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00265/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.818 sec, real 11.529 (submit 290.881 ms wait 11.230 s), input 127.818 MiB, send shutdown 2024/11/19-15:28:33.837957 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00265/att-02 2024/11/19-15:28:33.867260 139851179493120 [dcompact_etcd.cc:1773] job-00265/att-02: CleanFiles 0.029 sec : 0.000 2024/11/19-15:28:33.870267 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00265) = success 2024/11/19-15:28:33.870297 139851179493120 [dcompact_etcd.cc:1773] job-00265/att-03: CleanFiles 0.003 sec : 0.000 2024/11/19-15:28:33.870321 139851179493120 (Original Log Time 2024/11/19-15:27:59.249781) EVENT_LOG_v1 {"time_micros": 1732001279249773, "job": 265, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5323, 5326, 5329, 5333], "score": 7.95921, "input_data_size": 133300300, "oldest_snapshot_seqno": 153777508} 2024/11/19-15:28:33.870328 139851179493120 [compaction_job.cc:2437] [default] [JOB 265] Compacting 4@1 files to L2, score 7.96, subcompactions 1 : 1 2024/11/19-15:28:33.870336 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 263 Base level 1, inputs: [5323(31MB) 5326(31MB) 5329(31MB) 5333(31MB)] 2024/11/19-15:28:33.870760 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005447.sst) /storage/mytopling/..//tmp/Topling-82038-JG5NeV 2024/11/19-15:28:34.016969 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-02 2024/11/19-15:28:34.017065 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00270/att-02/compact.done) = No such file or directory: 2.646 sec on http probe failed: 5, giveup 2024/11/19-15:28:34.017084 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00270/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 24.862 sec, real 2.946 (submit 290.986 ms wait 2.646 s), input 24.862 MiB, send shutdown 2024/11/19-15:28:34.026058 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00270/att-02 2024/11/19-15:28:34.042497 139851148035840 [dcompact_etcd.cc:1773] job-00270/att-02: CleanFiles 0.016 sec : 0.000 2024/11/19-15:28:34.048319 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005447.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:28:34.100339 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005447.sst): index pass time = 0.05's, 142.673'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463573 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:28:34.171778 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005447.sst): first pass time = 0.30's, 222.959'MB/sec 2024/11/19-15:28:34.171834 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005447.sst): old prealloc_size = 73819750, real_size = 55624176 2024/11/19-15:28:34.172080 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005447.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463573 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.76 avg-zval = 117.85 2024/11/19-15:28:34.172133 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005447.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.124 sec, Key+Value = 0.067 GB 2024/11/19-15:28:34.246218 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00270) = success 2024/11/19-15:28:34.246255 139851148035840 [dcompact_etcd.cc:1773] job-00270/att-03: CleanFiles 0.204 sec : 0.000 2024/11/19-15:28:34.246298 139851148035840 (Original Log Time 2024/11/19-15:28:25.154170) EVENT_LOG_v1 {"time_micros": 1732001305154163, "job": 270, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5377], "score": 6.96061, "input_data_size": 25957763, "oldest_snapshot_seqno": -1} 2024/11/19-15:28:34.246314 139851148035840 [compaction_job.cc:2437] [default] [JOB 270] Compacting 1@1 files to L2, score 6.96, subcompactions 1 : 1 2024/11/19-15:28:34.246327 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 268 Base level 1, inputs: [5377(24MB)] 2024/11/19-15:28:34.246724 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005448.sst) /storage/mytopling/..//tmp/Topling-82038-aM1wYr 2024/11/19-15:28:34.286402 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005447.sst): second pass time = 0.00's,189926140.351'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.11's, 484.616'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3603.404'MB/sec (index lex order gen) rebuild zvType time = 0.06's, 2.030'MB/sec write SST data time = 0.06's, 970.130'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 189926140.351'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463573 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.76 avg-zval = 117.85 usrkeys = 463573 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.76 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6615 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5727 value = 0.9908 dict = 0.00 all = 1.2089 } Zip/UnZip{ index = 0.0946 value = 1.0093 dict = 0.00 all = 0.8272 } ---------------------------- total value len = 19.483522 GB avg = 0.268 KB (by entry num) total key len = 1.868983 GB avg = 0.026 KB total ukey len = 1.287427 GB avg = 0.018 KB total ukey num = 0.072694385 Billion total entry num = 0.072694385 Billion write speed all = 35.747284022 MB/sec (with seq num) write speed all = 34.773673890 MB/sec (without seq num) 2024/11/19-15:28:34.317181 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005448.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.026 GB 2024/11/19-15:28:34.330532 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005448.sst): index pass time = 0.01's, 216.329'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 180036 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0029 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:28:34.349225 139851179493120 [compaction_job.cc:1948] [default] [JOB 265] Generated table #5447: 463573 keys, 55513624 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:34.349283 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001314349258, "cf_name": "default", "job": 265, "event": "table_file_creation", "file_number": 5447, "file_size": 55513624, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54631200, "index_size": 876928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12980044, "raw_average_key_size": 28, "raw_value_size": 54128950, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463573, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001313, "oldest_key_time": 0, "file_creation_time": 1732001313, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5447, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:34.372006 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005449.sst) /storage/mytopling/..//tmp/Topling-82038-eCGB1w 2024/11/19-15:28:34.374599 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005448.sst): first pass time = 0.13's, 203.899'MB/sec 2024/11/19-15:28:34.374689 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005448.sst): old prealloc_size = 28553539, real_size = 21598640 2024/11/19-15:28:34.374890 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005448.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 180036 prefix = 4 raw-key = 0.0029 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0210 GB zip-val = 0.0212 GB avg-val = 116.80 avg-zval = 117.82 2024/11/19-15:28:34.374979 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005448.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.058 sec, Key+Value = 0.026 GB 2024/11/19-15:28:34.428624 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005448.sst): second pass time = 0.00's,71768518.771'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 399.624'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1774.637'MB/sec (index lex order gen) rebuild zvType time = 0.03's, 1.667'MB/sec write SST data time = 0.03's, 805.654'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.021'GB zip my value throughput = 71768518.771'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 180036 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 117.82 usrkeys = 180036 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 117.82 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 2574 UnZipSize{ index = 0.0036 GB value = 0.0210 GB dict = 0.00 MB all = 0.0261 GB } __ZipSize{ index = 0.0003 GB value = 0.0212 GB dict = 0.00 MB all = 0.0216 GB } UnZip/Zip{ index = 10.5437 value = 0.9913 dict = 0.00 all = 1.2093 } Zip/UnZip{ index = 0.0948 value = 1.0087 dict = 0.00 all = 0.8269 } ---------------------------- total value len = 19.504550 GB avg = 0.268 KB (by entry num) total key len = 1.874024 GB avg = 0.026 KB total ukey len = 1.291028 GB avg = 0.018 KB total ukey num = 0.072874421 Billion total entry num = 0.072874421 Billion write speed all = 35.782407775 MB/sec (with seq num) write speed all = 34.806618726 MB/sec (without seq num) 2024/11/19-15:28:34.434013 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-02 2024/11/19-15:28:34.434085 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00266/att-02/compact.done) = No such file or directory: 11.234 sec on http probe failed: 5, giveup 2024/11/19-15:28:34.434101 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00266/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.820 sec, real 11.525 (submit 283.071 ms wait 11.234 s), input 127.820 MiB, send shutdown 2024/11/19-15:28:34.437232 139851148035840 [compaction_job.cc:1948] [default] [JOB 270] Generated table #5448: 180036 keys, 21556376 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:34.437283 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001314437260, "cf_name": "default", "job": 270, "event": "table_file_creation", "file_number": 5448, "file_size": 21556376, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 21211936, "index_size": 341504, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 5041008, "raw_average_key_size": 28, "raw_value_size": 21028176, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 180036, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001314, "oldest_key_time": 0, "file_creation_time": 1732001314, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5448, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:34.441345 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00266/att-02 2024/11/19-15:28:34.443059 139851148035840 [compaction_job.cc:703] job-00270: subcompact[0], size: 0.021556 G, files: /storage/mytopling/.rocksdb [005448] 2024/11/19-15:28:34.452278 139851148035840 (Original Log Time 2024/11/19-15:28:34.246384) EVENT_LOG_v1 {"time_micros": 1732001314246350, "job": 270, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5377], "score": 6.96061, "input_data_size": 25957763, "oldest_snapshot_seqno": -1} 2024/11/19-15:28:34.452290 139851148035840 (Original Log Time 2024/11/19-15:28:34.451661) [compaction_job.cc:2022] [default] [JOB 270] Compacted 1@1 files to L2 => 21556376 bytes 2024/11/19-15:28:34.452295 139851148035840 (Original Log Time 2024/11/19-15:28:34.452139) [compaction_job.cc:1224] [default] compacted to: files[3 128 68 82 0 0 0] max score 6.43 (175 files need compaction), MB/sec: 132.0 rd, 109.6 wr, level 2, files in(1, 0) out(1 +0 blob) MB in(24.8, 0.0 +0.0 blob) out(20.6 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 180036, records dropped: 0 output_compression: Snappy 2024/11/19-15:28:34.452319 139851148035840 (Original Log Time 2024/11/19-15:28:34.452170) EVENT_LOG_v1 {"time_micros": 1732001314452156, "cf": "default", "job": 270, "event": "compaction_finished", "compaction_time_micros": 196686, "compaction_time_cpu_micros": 181105, "output_level": 2, "num_output_files": 1, "total_output_size": 21556376, "num_input_records": 180036, "num_output_records": 180036, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 128, 68, 82, 0, 0, 0]} 2024/11/19-15:28:34.452410 139851148035840 [compaction_job.cc:2437] [default] [JOB 275] Compacting 4@1 files to L2, score 6.43, subcompactions 1 : 1 2024/11/19-15:28:34.452418 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 273 Base level 1, inputs: [5265(31MB) 5268(31MB) 5270(31MB) 5272(31MB)] 2024/11/19-15:28:34.459423 139851137550080 [dcompact_etcd.cc:1773] job-00266/att-02: CleanFiles 0.018 sec : 0.000 2024/11/19-15:28:34.479327 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00266) = success 2024/11/19-15:28:34.479343 139851137550080 [dcompact_etcd.cc:1773] job-00266/att-03: CleanFiles 0.020 sec : 0.000 2024/11/19-15:28:34.479366 139851137550080 (Original Log Time 2024/11/19-15:27:59.842945) EVENT_LOG_v1 {"time_micros": 1732001279842938, "job": 266, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5336, 5339, 5342, 5344], "score": 7.70957, "input_data_size": 133300172, "oldest_snapshot_seqno": -1} 2024/11/19-15:28:34.479377 139851137550080 [compaction_job.cc:2437] [default] [JOB 266] Compacting 4@1 files to L2, score 7.71, subcompactions 1 : 1 2024/11/19-15:28:34.479388 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 264 Base level 1, inputs: [5336(31MB) 5339(31MB) 5342(31MB) 5344(31MB)] 2024/11/19-15:28:34.479634 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005450.sst) /storage/mytopling/..//tmp/Topling-82038-e11wOE 2024/11/19-15:28:34.587708 139834437928704 [db_impl_write.cc:2181] [default] New memtable created with log file: #5451. Immutable memtables: 0. 2024/11/19-15:28:34.588541 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005449.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:28:34.588725 139850967680768 [db_impl_compaction_flush.cc:103] [JOB 276] Syncing log #5440 2024/11/19-15:28:34.598592 139850967680768 (Original Log Time 2024/11/19-15:28:34.588367) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:28:34.598610 139850967680768 [flush_job.cc:879] [default] [JOB 276] Flushing memtable with next log file: 5451 2024/11/19-15:28:34.598638 139850967680768 EVENT_LOG_v1 {"time_micros": 1732001314598630, "job": 276, "event": "flush_started", "num_memtables": 1, "num_entries": 1577400, "num_deletes": 0, "total_data_size": 495208858, "memory_usage": 542899592, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:28:34.598645 139850967680768 [flush_job.cc:910] [default] [JOB 276] Level-0 flush table #5452: started 2024/11/19-15:28:34.599306 139850967680768 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000036.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/005452.sst in ConvertToSST, mem_size = 517.749 M, time = 646.320 us 2024/11/19-15:28:34.638570 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005449.sst): index pass time = 0.05's, 147.866'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462027 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:28:34.730284 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005449.sst): first pass time = 0.36's, 186.794'MB/sec 2024/11/19-15:28:34.730338 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005449.sst): old prealloc_size = 73819750, real_size = 55447264 2024/11/19-15:28:34.730654 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005449.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462027 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.84 avg-zval = 117.87 2024/11/19-15:28:34.730781 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005449.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.142 sec, Key+Value = 0.067 GB 2024/11/19-15:28:34.742560 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005450.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.001 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:28:34.752801 139851148035840 [dcompact_etcd.cc:1163] job-00275/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.300 sec, input = 126.920 MiB 2024/11/19-15:28:34.804496 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005450.sst): index pass time = 0.06's, 119.844'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463545 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:28:34.869145 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005449.sst): second pass time = 0.00's,194178338.129'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.14's, 398.700'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2643.926'MB/sec (index lex order gen) rebuild zvType time = 0.07's, 1.666'MB/sec write SST data time = 0.07's, 800.596'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 194178338.129'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462027 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.84 avg-zval = 117.87 usrkeys = 462027 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.84 avg-zval = 117.87 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6597 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0553 GB } UnZip/Zip{ index = 10.5690 value = 0.9913 dict = 0.00 all = 1.2093 } Zip/UnZip{ index = 0.0946 value = 1.0088 dict = 0.00 all = 0.8269 } ---------------------------- total value len = 19.558532 GB avg = 0.267 KB (by entry num) total key len = 1.886960 GB avg = 0.026 KB total ukey len = 1.300269 GB avg = 0.018 KB total ukey num = 0.073336448 Billion total entry num = 0.073336448 Billion write speed all = 35.867966104 MB/sec (with seq num) write speed all = 34.886714010 MB/sec (without seq num) 2024/11/19-15:28:34.874325 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-00 2024/11/19-15:28:34.874472 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00272/att-00/compact.done) = No such file or directory: 4.992 sec on http probe failed: 2, retry 2024/11/19-15:28:34.900023 139851179493120 [compaction_job.cc:1948] [default] [JOB 265] Generated table #5449: 462027 keys, 55337112 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:34.900103 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001314900067, "cf_name": "default", "job": 265, "event": "table_file_creation", "file_number": 5449, "file_size": 55337112, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54457296, "index_size": 874304, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12936756, "raw_average_key_size": 28, "raw_value_size": 53981578, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462027, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001314, "oldest_key_time": 0, "file_creation_time": 1732001314, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5449, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:34.905255 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005450.sst): first pass time = 0.43's, 157.680'MB/sec 2024/11/19-15:28:34.905338 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005450.sst): old prealloc_size = 73819750, real_size = 55621888 2024/11/19-15:28:34.905732 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005450.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463545 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.77 avg-zval = 117.85 2024/11/19-15:28:34.905817 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005450.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.163 sec, Key+Value = 0.067 GB 2024/11/19-15:28:34.920999 139851179493120 [compaction_job.cc:703] job-00265: subcompact[0], size: 0.110851 G, files: /storage/mytopling/.rocksdb [005447,005449] 2024/11/19-15:28:34.928949 139851179493120 (Original Log Time 2024/11/19-15:28:33.870381) EVENT_LOG_v1 {"time_micros": 1732001313870349, "job": 265, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5323, 5326, 5329, 5333], "score": 7.95921, "input_data_size": 133300300, "oldest_snapshot_seqno": 153777508} 2024/11/19-15:28:34.928960 139851179493120 (Original Log Time 2024/11/19-15:28:34.928274) [compaction_job.cc:2022] [default] [JOB 265] Compacted 4@1 files to L2 => 110850736 bytes 2024/11/19-15:28:34.928964 139851179493120 (Original Log Time 2024/11/19-15:28:34.928805) [compaction_job.cc:1224] [default] compacted to: files[3 124 70 82 0 0 0] max score 6.18 (173 files need compaction), MB/sec: 126.9 rd, 105.5 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925600, records dropped: 0 output_compression: Snappy 2024/11/19-15:28:34.928980 139851179493120 (Original Log Time 2024/11/19-15:28:34.928833) EVENT_LOG_v1 {"time_micros": 1732001314928819, "cf": "default", "job": 265, "event": "compaction_finished", "compaction_time_micros": 1050630, "compaction_time_cpu_micros": 901371, "output_level": 2, "num_output_files": 2, "total_output_size": 110850736, "num_input_records": 925600, "num_output_records": 925600, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [3, 124, 70, 82, 0, 0, 0]} 2024/11/19-15:28:34.967438 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005251.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:34.967480 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001314967476, "job": 265, "event": "table_file_deletion", "file_number": 5251} 2024/11/19-15:28:34.984481 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005288.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:34.984528 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001314984522, "job": 265, "event": "table_file_deletion", "file_number": 5288} 2024/11/19-15:28:34.991489 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005289.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:34.991529 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001314991524, "job": 265, "event": "table_file_deletion", "file_number": 5289} 2024/11/19-15:28:34.991809 139851179493120 [compaction_job.cc:2437] [default] [JOB 277] Compacting 4@1 files to L2, score 6.18, subcompactions 1 : 1 2024/11/19-15:28:34.991825 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 274 Base level 1, inputs: [5275(31MB) 5277(31MB) 5278(31MB) 5281(31MB)] 2024/11/19-15:28:35.061355 139850967680768 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/005452.sst): fsize = 517.751 M, time(ms): open: 0.044, seek: 0.018, finish: 0.049, meta: 0.020, Flush: 0.015, sync: 461.871, close: 0.009, all: 462.026 2024/11/19-15:28:35.062545 139850967680768 [flush_job.cc:1083] [default] [JOB 276] Flush lasted 463956 microseconds, and 250685 cpu microseconds. 2024/11/19-15:28:35.085355 139850967680768 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:28:35.085411 139850967680768 (Original Log Time 2024/11/19-15:28:35.061397) [flush_job.cc:1029] [default] [JOB 276] Level-0 flush table #5452: 542901018 bytes OK (needs compaction) 2024/11/19-15:28:35.085417 139850967680768 (Original Log Time 2024/11/19-15:28:35.062563) [memtable_list.cc:560] [default] Level-0 commit table #5452 started 2024/11/19-15:28:35.085421 139850967680768 (Original Log Time 2024/11/19-15:28:35.085323) [memtable_list.cc:759] [default] Level-0 commit table #5452: memtable #1 done 2024/11/19-15:28:35.085423 139850967680768 (Original Log Time 2024/11/19-15:28:35.085349) EVENT_LOG_v1 {"time_micros": 1732001315085343, "job": 276, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [4, 124, 70, 82, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:28:35.085426 139850967680768 (Original Log Time 2024/11/19-15:28:35.085383) [db_impl_compaction_flush.cc:310] [default] Level summary: files[4 124 70 82 0 0 0] max score 5.93 (170 files need compaction) 2024/11/19-15:28:35.085440 139850967680768 [db_impl_files.cc:470] [JOB 276] Try to delete WAL files size 485824511, prev total WAL file size 986745819, number of live WAL files 4. 2024/11/19-15:28:35.089608 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005450.sst): second pass time = 0.00's,206601839.695'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.18's, 301.248'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2239.415'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 1.232'MB/sec write SST data time = 0.09's, 618.144'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 206601839.695'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463545 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 117.85 usrkeys = 463545 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6615 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5720 value = 0.9909 dict = 0.00 all = 1.2089 } Zip/UnZip{ index = 0.0946 value = 1.0092 dict = 0.00 all = 0.8272 } ---------------------------- total value len = 19.612661 GB avg = 0.266 KB (by entry num) total key len = 1.899940 GB avg = 0.026 KB total ukey len = 1.309540 GB avg = 0.018 KB total ukey num = 0.073799993 Billion total entry num = 0.073799993 Billion write speed all = 35.966944745 MB/sec (with seq num) write speed all = 34.979854328 MB/sec (without seq num) 2024/11/19-15:28:35.103128 139851137550080 [compaction_job.cc:1948] [default] [JOB 266] Generated table #5450: 463545 keys, 55511320 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:35.103191 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001315103159, "cf_name": "default", "job": 266, "event": "table_file_creation", "file_number": 5450, "file_size": 55511320, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54628912, "index_size": 876928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12979260, "raw_average_key_size": 28, "raw_value_size": 54129682, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463545, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001314, "oldest_key_time": 0, "file_creation_time": 1732001314, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5450, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:35.119473 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005453.sst) /storage/mytopling/..//tmp/Topling-82038-rcWgfQ 2024/11/19-15:28:35.153948 139850967680768 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005426.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:35.293278 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005453.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:28:35.305997 139851179493120 [dcompact_etcd.cc:1163] job-00277/att-00: init time wrt = 0.001, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.313 sec, input = 126.930 MiB 2024/11/19-15:28:35.330041 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005453.sst): index pass time = 0.04's, 201.307'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462101 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:28:35.399628 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-02 2024/11/19-15:28:35.399748 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00267/att-02/compact.done) = No such file or directory: 11.229 sec on http probe failed: 5, giveup 2024/11/19-15:28:35.399776 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00267/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.822 sec, real 11.516 (submit 280.075 ms wait 11.229 s), input 127.822 MiB, send shutdown 2024/11/19-15:28:35.403735 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005453.sst): first pass time = 0.28's, 235.429'MB/sec 2024/11/19-15:28:35.403780 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005453.sst): old prealloc_size = 73819750, real_size = 55459280 2024/11/19-15:28:35.404021 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005453.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462101 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.82 avg-zval = 117.86 2024/11/19-15:28:35.404078 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005453.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.111 sec, Key+Value = 0.067 GB 2024/11/19-15:28:35.408495 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00267/att-02 2024/11/19-15:28:35.408739 139851127064320 [dcompact_etcd.cc:1773] job-00267/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:35.410659 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00267) = success 2024/11/19-15:28:35.410671 139851127064320 [dcompact_etcd.cc:1773] job-00267/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:28:35.410697 139851127064320 (Original Log Time 2024/11/19-15:28:00.818601) EVENT_LOG_v1 {"time_micros": 1732001280818590, "job": 267, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5346, 5350, 5355, 5359], "score": 7.45992, "input_data_size": 133300044, "oldest_snapshot_seqno": -1} 2024/11/19-15:28:35.410708 139851127064320 [compaction_job.cc:2437] [default] [JOB 267] Compacting 4@1 files to L2, score 7.46, subcompactions 1 : 1 2024/11/19-15:28:35.410719 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 265 Base level 1, inputs: [5346(31MB) 5350(31MB) 5355(31MB) 5359(31MB)] 2024/11/19-15:28:35.410980 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005454.sst) /storage/mytopling/..//tmp/Topling-82038-VKYCvT 2024/11/19-15:28:35.522952 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005453.sst): second pass time = 0.00's,209227682.171'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 464.538'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3687.119'MB/sec (index lex order gen) rebuild zvType time = 0.06's, 1.782'MB/sec write SST data time = 0.05's, 1023.708'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 209227682.171'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462101 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.82 avg-zval = 117.86 usrkeys = 462101 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.82 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6596 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0553 GB } UnZip/Zip{ index = 10.5038 value = 0.9911 dict = 0.00 all = 1.2090 } Zip/UnZip{ index = 0.0952 value = 1.0089 dict = 0.00 all = 0.8271 } ---------------------------- total value len = 19.666642 GB avg = 0.265 KB (by entry num) total key len = 1.912878 GB avg = 0.026 KB total ukey len = 1.318782 GB avg = 0.018 KB total ukey num = 0.074262094 Billion total entry num = 0.074262094 Billion write speed all = 36.052708050 MB/sec (with seq num) write speed all = 35.060156023 MB/sec (without seq num) 2024/11/19-15:28:35.527258 139851137550080 [compaction_job.cc:1948] [default] [JOB 266] Generated table #5453: 462101 keys, 55349016 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:35.527310 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001315527286, "cf_name": "default", "job": 266, "event": "table_file_creation", "file_number": 5453, "file_size": 55349016, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54463680, "index_size": 879872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12938828, "raw_average_key_size": 28, "raw_value_size": 53980742, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462101, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001315, "oldest_key_time": 0, "file_creation_time": 1732001315, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5453, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:35.542111 139851137550080 [compaction_job.cc:703] job-00266: subcompact[0], size: 0.110860 G, files: /storage/mytopling/.rocksdb [005450,005453] 2024/11/19-15:28:35.543483 139851137550080 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:28:35.543611 139851137550080 (Original Log Time 2024/11/19-15:28:34.479425) EVENT_LOG_v1 {"time_micros": 1732001314479401, "job": 266, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5336, 5339, 5342, 5344], "score": 7.70957, "input_data_size": 133300172, "oldest_snapshot_seqno": -1} 2024/11/19-15:28:35.543618 139851137550080 (Original Log Time 2024/11/19-15:28:35.543070) [compaction_job.cc:2022] [default] [JOB 266] Compacted 4@1 files to L2 => 110860336 bytes 2024/11/19-15:28:35.543621 139851137550080 (Original Log Time 2024/11/19-15:28:35.543449) [compaction_job.cc:1224] [default] compacted to: files[4 120 72 82 0 0 0] max score 5.93 (172 files need compaction), MB/sec: 125.4 rd, 104.3 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925646, records dropped: 0 output_compression: Snappy 2024/11/19-15:28:35.543642 139851137550080 (Original Log Time 2024/11/19-15:28:35.543476) EVENT_LOG_v1 {"time_micros": 1732001315543462, "cf": "default", "job": 266, "event": "compaction_finished", "compaction_time_micros": 1062699, "compaction_time_cpu_micros": 986567, "output_level": 2, "num_output_files": 2, "total_output_size": 110860336, "num_input_records": 925646, "num_output_records": 925646, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 120, 72, 82, 0, 0, 0]} 2024/11/19-15:28:35.547709 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005267.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:35.547733 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001315547730, "job": 266, "event": "table_file_deletion", "file_number": 5267} 2024/11/19-15:28:35.552035 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005269.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:35.552057 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001315552055, "job": 266, "event": "table_file_deletion", "file_number": 5269} 2024/11/19-15:28:35.556633 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005271.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:35.556654 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001315556651, "job": 266, "event": "table_file_deletion", "file_number": 5271} 2024/11/19-15:28:35.561331 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005273.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:35.561350 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001315561348, "job": 266, "event": "table_file_deletion", "file_number": 5273} 2024/11/19-15:28:35.561501 139851137550080 [compaction_job.cc:2437] [default] [JOB 278] Compacting 4@1 files to L2, score 5.93, subcompactions 1 : 1 2024/11/19-15:28:35.561510 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 276 Base level 1, inputs: [5283(31MB) 5285(31MB) 5287(31MB) 5290(31MB)] 2024/11/19-15:28:35.584859 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005454.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:28:35.618788 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005454.sst): index pass time = 0.03's, 218.696'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463432 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:28:35.691187 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005454.sst): first pass time = 0.28's, 239.510'MB/sec 2024/11/19-15:28:35.691239 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005454.sst): old prealloc_size = 73819750, real_size = 55612400 2024/11/19-15:28:35.691478 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005454.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463432 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.81 avg-zval = 117.86 2024/11/19-15:28:35.691525 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005454.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.107 sec, Key+Value = 0.067 GB 2024/11/19-15:28:35.787100 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005454.sst): second pass time = 0.00's,201987940.299'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 579.078'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3721.161'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.586'MB/sec write SST data time = 0.05's, 1092.363'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 201987940.299'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463432 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 117.86 usrkeys = 463432 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 117.86 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6616 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.5725 value = 0.9911 dict = 0.00 all = 1.2091 } Zip/UnZip{ index = 0.0946 value = 1.0090 dict = 0.00 all = 0.8270 } ---------------------------- total value len = 19.720775 GB avg = 0.264 KB (by entry num) total key len = 1.925854 GB avg = 0.026 KB total ukey len = 1.328050 GB avg = 0.018 KB total ukey num = 0.074725526 Billion total entry num = 0.074725526 Billion write speed all = 36.148872874 MB/sec (with seq num) write speed all = 35.150567409 MB/sec (without seq num) 2024/11/19-15:28:35.791093 139851127064320 [compaction_job.cc:1948] [default] [JOB 267] Generated table #5454: 463432 keys, 55501848 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:35.791149 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001315791121, "cf_name": "default", "job": 267, "event": "table_file_creation", "file_number": 5454, "file_size": 55501848, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54619680, "index_size": 876672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12976096, "raw_average_key_size": 28, "raw_value_size": 54132768, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463432, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001315, "oldest_key_time": 0, "file_creation_time": 1732001315, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5454, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:35.806090 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005455.sst) /storage/mytopling/..//tmp/Topling-82038-BLbw1C 2024/11/19-15:28:35.820352 139851137550080 [dcompact_etcd.cc:1163] job-00278/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.258 sec, input = 126.908 MiB 2024/11/19-15:28:35.824810 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-00 2024/11/19-15:28:35.824859 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00274/att-00/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:28:35.946566 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005455.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:28:35.980705 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005455.sst): index pass time = 0.03's, 216.868'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462236 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 2024/11/19-15:28:36.050040 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005455.sst): first pass time = 0.24's, 274.347'MB/sec 2024/11/19-15:28:36.050094 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005455.sst): old prealloc_size = 73819750, real_size = 55471360 2024/11/19-15:28:36.050324 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005455.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462236 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.78 avg-zval = 117.85 2024/11/19-15:28:36.050382 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005455.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.104 sec, Key+Value = 0.067 GB 2024/11/19-15:28:36.145508 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005455.sst): second pass time = 0.00's,188740377.622'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.10's, 580.317'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3903.020'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.573'MB/sec write SST data time = 0.05's, 1101.399'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 188740377.622'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462236 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.78 avg-zval = 117.85 usrkeys = 462236 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.78 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6597 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0554 GB } UnZip/Zip{ index = 10.5023 value = 0.9909 dict = 0.00 all = 1.2088 } Zip/UnZip{ index = 0.0952 value = 1.0092 dict = 0.00 all = 0.8272 } ---------------------------- total value len = 19.774755 GB avg = 0.263 KB (by entry num) total key len = 1.938797 GB avg = 0.026 KB total ukey len = 1.337295 GB avg = 0.018 KB total ukey num = 0.075187762 Billion total entry num = 0.075187762 Billion write speed all = 36.238940156 MB/sec (with seq num) write speed all = 35.235060238 MB/sec (without seq num) 2024/11/19-15:28:36.149263 139851127064320 [compaction_job.cc:1948] [default] [JOB 267] Generated table #5455: 462236 keys, 55361112 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:36.149316 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001316149292, "cf_name": "default", "job": 267, "event": "table_file_creation", "file_number": 5455, "file_size": 55361112, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54475376, "index_size": 880256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12942608, "raw_average_key_size": 28, "raw_value_size": 53979748, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462236, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001315, "oldest_key_time": 0, "file_creation_time": 1732001315, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5455, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:36.163757 139851127064320 [compaction_job.cc:703] job-00267: subcompact[0], size: 0.110863 G, files: /storage/mytopling/.rocksdb [005454,005455] 2024/11/19-15:28:36.165103 139851127064320 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:28:36.165203 139851127064320 (Original Log Time 2024/11/19-15:28:35.410754) EVENT_LOG_v1 {"time_micros": 1732001315410734, "job": 267, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5346, 5350, 5355, 5359], "score": 7.45992, "input_data_size": 133300044, "oldest_snapshot_seqno": -1} 2024/11/19-15:28:36.165208 139851127064320 (Original Log Time 2024/11/19-15:28:36.164658) [compaction_job.cc:2022] [default] [JOB 267] Compacted 4@1 files to L2 => 110862960 bytes 2024/11/19-15:28:36.165212 139851127064320 (Original Log Time 2024/11/19-15:28:36.165070) [compaction_job.cc:1224] [default] compacted to: files[4 116 74 82 0 0 0] max score 5.68 (170 files need compaction), MB/sec: 177.0 rd, 147.2 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925668, records dropped: 0 output_compression: Snappy 2024/11/19-15:28:36.165231 139851127064320 (Original Log Time 2024/11/19-15:28:36.165096) EVENT_LOG_v1 {"time_micros": 1732001316165082, "cf": "default", "job": 267, "event": "compaction_finished", "compaction_time_micros": 753016, "compaction_time_cpu_micros": 750031, "output_level": 2, "num_output_files": 2, "total_output_size": 110862960, "num_input_records": 925668, "num_output_records": 925668, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 116, 74, 82, 0, 0, 0]} 2024/11/19-15:28:36.169114 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005276.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:36.169140 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001316169137, "job": 267, "event": "table_file_deletion", "file_number": 5276} 2024/11/19-15:28:36.173095 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005279.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:36.173118 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001316173115, "job": 267, "event": "table_file_deletion", "file_number": 5279} 2024/11/19-15:28:36.177298 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005282.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:36.177320 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001316177318, "job": 267, "event": "table_file_deletion", "file_number": 5282} 2024/11/19-15:28:36.182267 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005284.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:36.182286 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001316182284, "job": 267, "event": "table_file_deletion", "file_number": 5284} 2024/11/19-15:28:36.182425 139851127064320 [compaction_job.cc:2437] [default] [JOB 279] Compacting 4@1 files to L2, score 5.68, subcompactions 1 : 1 2024/11/19-15:28:36.182433 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 277 Base level 1, inputs: [5291(31MB) 5292(31MB) 5294(31MB) 5295(31MB)] 2024/11/19-15:28:36.288947 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-02 2024/11/19-15:28:36.289048 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00268/att-02/compact.done) = No such file or directory: 3.196 sec on http probe failed: 1, retry 2024/11/19-15:28:36.465642 139851127064320 [dcompact_etcd.cc:1163] job-00279/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.283 sec, input = 126.846 MiB 2024/11/19-15:28:36.879646 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-00 2024/11/19-15:28:36.879740 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00272/att-00/compact.done) = No such file or directory: 6.997 sec on http probe failed: 3, retry 2024/11/19-15:28:37.833828 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-00 2024/11/19-15:28:37.833903 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00274/att-00/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:28:37.934416 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-00 2024/11/19-15:28:37.934506 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00275/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:28:38.297855 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-02 2024/11/19-15:28:38.297937 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00268/att-02/compact.done) = No such file or directory: 5.203 sec on http probe failed: 2, retry 2024/11/19-15:28:38.487148 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-00 2024/11/19-15:28:38.487230 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00277/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:28:38.891224 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-00 2024/11/19-15:28:38.891329 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00272/att-00/compact.done) = No such file or directory: 9.002 sec on http probe failed: 4, retry 2024/11/19-15:28:38.998496 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-00 2024/11/19-15:28:38.998581 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00278/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:28:39.646635 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-00 2024/11/19-15:28:39.646721 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00279/att-00/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:28:39.840070 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-00 2024/11/19-15:28:39.840146 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00274/att-00/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:28:39.942747 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-00 2024/11/19-15:28:39.942841 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00275/att-00/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:28:40.307288 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-02 2024/11/19-15:28:40.307393 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00268/att-02/compact.done) = No such file or directory: 7.212 sec on http probe failed: 3, retry 2024/11/19-15:28:40.494827 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-00 2024/11/19-15:28:40.494980 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00277/att-00/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:28:40.901896 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-00 2024/11/19-15:28:40.902016 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00272/att-00/compact.done) = No such file or directory: 11.014 sec on http probe failed: 5, giveup 2024/11/19-15:28:40.902043 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00272/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 119.253 sec, real 11.303 (submit 278.024 ms wait 11.014 s), input 119.253 MiB, send shutdown 2024/11/19-15:28:40.910058 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-00 2024/11/19-15:28:40.910429 139851158521600 [dcompact_etcd.cc:1773] job-00272/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:41.006285 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-00 2024/11/19-15:28:41.006394 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00278/att-00/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:28:41.193602 139851158521600 [dcompact_etcd.cc:1163] job-00272/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.283 sec, input = 119.253 MiB 2024/11/19-15:28:41.656369 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-00 2024/11/19-15:28:41.656448 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00279/att-00/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:28:41.849062 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-00 2024/11/19-15:28:41.849143 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00274/att-00/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:28:41.849163 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00274/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.907 sec, real 11.516 (submit 304.816 ms wait 11.202 s), input 126.907 MiB, send shutdown 2024/11/19-15:28:41.854133 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-00 2024/11/19-15:28:41.854356 139850978166528 [dcompact_etcd.cc:1773] job-00274/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:41.948080 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-00 2024/11/19-15:28:41.948150 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00275/att-00/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:28:42.119995 139850978166528 [dcompact_etcd.cc:1163] job-00274/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.266 sec, input = 126.907 MiB 2024/11/19-15:28:42.313545 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-02 2024/11/19-15:28:42.313623 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00268/att-02/compact.done) = No such file or directory: 9.221 sec on http probe failed: 4, retry 2024/11/19-15:28:42.504310 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-00 2024/11/19-15:28:42.504422 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00277/att-00/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:28:43.014928 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-00 2024/11/19-15:28:43.015026 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00278/att-00/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:28:43.615886 139852362282752 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:28:43.665086 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-00 2024/11/19-15:28:43.665178 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00279/att-00/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:28:43.953508 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-00 2024/11/19-15:28:43.953580 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00275/att-00/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:28:44.183826 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-01 2024/11/19-15:28:44.183940 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00272/att-01/compact.done) = No such file or directory: 2.981 sec on http probe failed: 1, retry 2024/11/19-15:28:44.322329 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-02 2024/11/19-15:28:44.322396 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00268/att-02/compact.done) = No such file or directory: 11.227 sec on http probe failed: 5, giveup 2024/11/19-15:28:44.322414 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00268/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 127.823 sec, real 11.527 (submit 290.993 ms wait 11.227 s), input 127.823 MiB, send shutdown 2024/11/19-15:28:44.330224 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00268/att-02 2024/11/19-15:28:44.330474 139851169007360 [dcompact_etcd.cc:1773] job-00268/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:44.332321 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00268) = success 2024/11/19-15:28:44.332331 139851169007360 [dcompact_etcd.cc:1773] job-00268/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:28:44.332349 139851169007360 (Original Log Time 2024/11/19-15:28:09.736659) EVENT_LOG_v1 {"time_micros": 1732001289736651, "job": 268, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5364, 5367, 5371, 5374], "score": 7.21027, "input_data_size": 133300236, "oldest_snapshot_seqno": -1} 2024/11/19-15:28:44.332357 139851169007360 [compaction_job.cc:2437] [default] [JOB 268] Compacting 4@1 files to L2, score 7.21, subcompactions 1 : 1 2024/11/19-15:28:44.332364 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 266 Base level 1, inputs: [5364(31MB) 5367(31MB) 5371(31MB) 5374(31MB)] 2024/11/19-15:28:44.332582 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005456.sst) /storage/mytopling/..//tmp/Topling-82038-Cf2sr5 2024/11/19-15:28:44.472892 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005456.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:28:44.509380 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005456.sst): index pass time = 0.04's, 203.406'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 463354 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.92 2024/11/19-15:28:44.509834 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-00 2024/11/19-15:28:44.509939 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00277/att-00/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:28:44.576079 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005456.sst): first pass time = 0.24's, 275.620'MB/sec 2024/11/19-15:28:44.576134 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005456.sst): old prealloc_size = 73819750, real_size = 55617392 2024/11/19-15:28:44.576375 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005456.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 463354 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0541 GB zip-val = 0.0546 GB avg-val = 116.83 avg-zval = 117.87 2024/11/19-15:28:44.576440 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005456.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.104 sec, Key+Value = 0.067 GB 2024/11/19-15:28:44.670342 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005456.sst): second pass time = 0.00's,179254980.132'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 589.299'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3734.107'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.606'MB/sec write SST data time = 0.05's, 1121.075'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 179254980.132'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 463354 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.83 avg-zval = 117.87 usrkeys = 463354 avg-key = 20.00 avg-zkey = 1.92 avg-val = 116.83 avg-zval = 117.87 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6616 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0546 GB dict = 0.00 MB all = 0.0555 GB } UnZip/Zip{ index = 10.4351 value = 0.9912 dict = 0.00 all = 1.2090 } Zip/UnZip{ index = 0.0958 value = 1.0088 dict = 0.00 all = 0.8271 } ---------------------------- total value len = 19.828890 GB avg = 0.262 KB (by entry num) total key len = 1.951771 GB avg = 0.026 KB total ukey len = 1.346562 GB avg = 0.018 KB total ukey num = 0.075651116 Billion total entry num = 0.075651116 Billion write speed all = 35.841011828 MB/sec (with seq num) write speed all = 34.845114560 MB/sec (without seq num) 2024/11/19-15:28:44.674041 139851169007360 [compaction_job.cc:1948] [default] [JOB 268] Generated table #5456: 463354 keys, 55506840 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:44.674092 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001324674068, "cf_name": "default", "job": 268, "event": "table_file_creation", "file_number": 5456, "file_size": 55506840, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54613280, "index_size": 888064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12973912, "raw_average_key_size": 28, "raw_value_size": 54135004, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 463354, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001324, "oldest_key_time": 0, "file_creation_time": 1732001324, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5456, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:44.687781 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005457.sst) /storage/mytopling/..//tmp/Topling-82038-qAnR88 2024/11/19-15:28:44.826669 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005457.sst: sumWaitingMem = 0.006 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0060 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:28:44.861439 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005457.sst): index pass time = 0.03's, 212.983'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 462327 min-keylen = 16 max-keylen = 16 prefix = 4 subcplen = 3 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 2024/11/19-15:28:44.929015 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005457.sst): first pass time = 0.24's, 277.440'MB/sec 2024/11/19-15:28:44.929068 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005457.sst): old prealloc_size = 73819750, real_size = 55473152 2024/11/19-15:28:44.929309 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005457.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = MixedLenBlobStore usrkeys = 462327 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0540 GB zip-val = 0.0545 GB avg-val = 116.75 avg-zval = 117.85 2024/11/19-15:28:44.929378 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005457.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.103 sec, Key+Value = 0.067 GB 2024/11/19-15:28:45.023812 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005457.sst): second pass time = 0.00's,192780592.857'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 584.423'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3656.302'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 2.592'MB/sec write SST data time = 0.05's, 1109.276'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.054'GB zip my value throughput = 192780592.857'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 462327 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.75 avg-zval = 117.85 usrkeys = 462327 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.75 avg-zval = 117.85 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6597 UnZipSize{ index = 0.0092 GB value = 0.0540 GB dict = 0.00 MB all = 0.0669 GB } __ZipSize{ index = 0.0009 GB value = 0.0545 GB dict = 0.00 MB all = 0.0554 GB } UnZip/Zip{ index = 10.5767 value = 0.9907 dict = 0.00 all = 1.2088 } Zip/UnZip{ index = 0.0945 value = 1.0093 dict = 0.00 all = 0.8273 } ---------------------------- total value len = 19.882868 GB avg = 0.261 KB (by entry num) total key len = 1.964716 GB avg = 0.026 KB total ukey len = 1.355809 GB avg = 0.018 KB total ukey num = 0.076113443 Billion total entry num = 0.076113443 Billion write speed all = 35.930238636 MB/sec (with seq num) write speed all = 34.928837608 MB/sec (without seq num) 2024/11/19-15:28:45.025247 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-00 2024/11/19-15:28:45.025330 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00278/att-00/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:28:45.027560 139851169007360 [compaction_job.cc:1948] [default] [JOB 268] Generated table #5457: 462327 keys, 55362840 bytes (need compaction), temperature: kUnknown 2024/11/19-15:28:45.027609 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001325027586, "cf_name": "default", "job": 268, "event": "table_file_creation", "file_number": 5457, "file_size": 55362840, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 54483120, "index_size": 874240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12945156, "raw_average_key_size": 28, "raw_value_size": 53978566, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 462327, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-MixedLenBlobStore", "creation_time": 1732001324, "oldest_key_time": 0, "file_creation_time": 1732001324, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5457, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:28:45.041195 139851169007360 [compaction_job.cc:703] job-00268: subcompact[0], size: 0.110870 G, files: /storage/mytopling/.rocksdb [005456,005457] 2024/11/19-15:28:45.042496 139851169007360 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:28:45.042603 139851169007360 (Original Log Time 2024/11/19-15:28:44.332390) EVENT_LOG_v1 {"time_micros": 1732001324332373, "job": 268, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5364, 5367, 5371, 5374], "score": 7.21027, "input_data_size": 133300236, "oldest_snapshot_seqno": -1} 2024/11/19-15:28:45.042609 139851169007360 (Original Log Time 2024/11/19-15:28:45.042094) [compaction_job.cc:2022] [default] [JOB 268] Compacted 4@1 files to L2 => 110869680 bytes 2024/11/19-15:28:45.042612 139851169007360 (Original Log Time 2024/11/19-15:28:45.042464) [compaction_job.cc:1224] [default] compacted to: files[4 112 76 82 0 0 0] max score 5.43 (168 files need compaction), MB/sec: 188.1 rd, 156.4 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.1, 0.0 +0.0 blob) out(105.7 +0.0 blob), read-write-amplify(1.8) write-amplify(0.8) OK, records in: 925681, records dropped: 0 output_compression: Snappy 2024/11/19-15:28:45.042630 139851169007360 (Original Log Time 2024/11/19-15:28:45.042489) EVENT_LOG_v1 {"time_micros": 1732001325042477, "cf": "default", "job": 268, "event": "compaction_finished", "compaction_time_micros": 708817, "compaction_time_cpu_micros": 705999, "output_level": 2, "num_output_files": 2, "total_output_size": 110869680, "num_input_records": 925681, "num_output_records": 925681, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 112, 76, 82, 0, 0, 0]} 2024/11/19-15:28:45.046718 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005274.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:45.046742 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001325046739, "job": 268, "event": "table_file_deletion", "file_number": 5274} 2024/11/19-15:28:45.050549 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005280.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:45.050569 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001325050567, "job": 268, "event": "table_file_deletion", "file_number": 5280} 2024/11/19-15:28:45.054542 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005286.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:45.054563 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001325054561, "job": 268, "event": "table_file_deletion", "file_number": 5286} 2024/11/19-15:28:45.059014 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005293.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:45.059036 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001325059034, "job": 268, "event": "table_file_deletion", "file_number": 5293} 2024/11/19-15:28:45.064372 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005297.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:45.064393 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001325064391, "job": 268, "event": "table_file_deletion", "file_number": 5297} 2024/11/19-15:28:45.069094 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005300.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:45.069114 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001325069112, "job": 268, "event": "table_file_deletion", "file_number": 5300} 2024/11/19-15:28:45.073719 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005303.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:45.073738 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001325073736, "job": 268, "event": "table_file_deletion", "file_number": 5303} 2024/11/19-15:28:45.077846 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005305.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:45.077865 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001325077863, "job": 268, "event": "table_file_deletion", "file_number": 5305} 2024/11/19-15:28:45.078012 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005441.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:28:45.078022 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001325078021, "job": 268, "event": "table_file_deletion", "file_number": 5441} 2024/11/19-15:28:45.078168 139851169007360 [compaction_job.cc:2437] [default] [JOB 280] Compacting 4@1 files to L2, score 5.43, subcompactions 1 : 1 2024/11/19-15:28:45.078177 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 278 Base level 1, inputs: [5296(31MB) 5298(31MB) 5299(31MB) 5301(31MB)] 2024/11/19-15:28:45.301831 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-01 2024/11/19-15:28:45.301940 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00274/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:28:45.372341 139851169007360 [dcompact_etcd.cc:1163] job-00280/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.294 sec, input = 126.971 MiB 2024/11/19-15:28:45.674769 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-00 2024/11/19-15:28:45.674898 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00279/att-00/compact.done) = No such file or directory: 9.200 sec on http probe failed: 4, retry 2024/11/19-15:28:45.961525 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-00 2024/11/19-15:28:45.961617 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00275/att-00/compact.done) = No such file or directory: 11.201 sec on http probe failed: 5, giveup 2024/11/19-15:28:45.961636 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00275/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.920 sec, real 11.509 (submit 299.950 ms wait 11.201 s), input 126.920 MiB, send shutdown 2024/11/19-15:28:45.967062 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-00 2024/11/19-15:28:45.967289 139851148035840 [dcompact_etcd.cc:1773] job-00275/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:46.188979 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-01 2024/11/19-15:28:46.189089 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00272/att-01/compact.done) = No such file or directory: 4.990 sec on http probe failed: 2, retry 2024/11/19-15:28:46.211740 139851148035840 [dcompact_etcd.cc:1163] job-00275/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.244 sec, input = 126.920 MiB 2024/11/19-15:28:46.519307 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-00 2024/11/19-15:28:46.519409 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00277/att-00/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:28:46.519427 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00277/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.930 sec, real 11.527 (submit 313.396 ms wait 11.204 s), input 126.930 MiB, send shutdown 2024/11/19-15:28:46.523819 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-00 2024/11/19-15:28:46.524063 139851179493120 [dcompact_etcd.cc:1773] job-00277/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:46.802588 139851179493120 [dcompact_etcd.cc:1163] job-00277/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.278 sec, input = 126.930 MiB 2024/11/19-15:28:47.035042 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-00 2024/11/19-15:28:47.035165 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00278/att-00/compact.done) = No such file or directory: 11.205 sec on http probe failed: 5, giveup 2024/11/19-15:28:47.035190 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00278/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.908 sec, real 11.473 (submit 258.426 ms wait 11.205 s), input 126.908 MiB, send shutdown 2024/11/19-15:28:47.043846 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-00 2024/11/19-15:28:47.044128 139851137550080 [dcompact_etcd.cc:1773] job-00278/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:47.310041 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-01 2024/11/19-15:28:47.310138 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00274/att-01/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:28:47.335326 139851137550080 [dcompact_etcd.cc:1163] job-00278/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.291 sec, input = 126.908 MiB 2024/11/19-15:28:47.684616 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-00 2024/11/19-15:28:47.684717 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00279/att-00/compact.done) = No such file or directory: 11.209 sec on http probe failed: 5, giveup 2024/11/19-15:28:47.684737 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00279/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.846 sec, real 11.502 (submit 282.793 ms wait 11.209 s), input 126.846 MiB, send shutdown 2024/11/19-15:28:47.693605 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-00 2024/11/19-15:28:47.693845 139851127064320 [dcompact_etcd.cc:1773] job-00279/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:48.003961 139851127064320 [dcompact_etcd.cc:1163] job-00279/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.310 sec, input = 126.846 MiB 2024/11/19-15:28:48.194151 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-01 2024/11/19-15:28:48.194254 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00272/att-01/compact.done) = No such file or directory: 6.996 sec on http probe failed: 3, retry 2024/11/19-15:28:48.555168 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-00 2024/11/19-15:28:48.555268 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00280/att-00/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:28:49.315972 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-01 2024/11/19-15:28:49.316071 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00274/att-01/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:28:49.394689 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-01 2024/11/19-15:28:49.394768 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00275/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:28:49.984765 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-01 2024/11/19-15:28:49.984865 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00277/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:28:50.205698 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-01 2024/11/19-15:28:50.205802 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00272/att-01/compact.done) = No such file or directory: 9.001 sec on http probe failed: 4, retry 2024/11/19-15:28:50.516630 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-01 2024/11/19-15:28:50.516716 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00278/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:28:50.560314 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-00 2024/11/19-15:28:50.560391 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00280/att-00/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:28:51.186658 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-01 2024/11/19-15:28:51.186776 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00279/att-01/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:28:51.319073 139852349699840 [db_impl_write.cc:2181] [default] New memtable created with log file: #5458. Immutable memtables: 0. 2024/11/19-15:28:51.319535 139852349699840 [column_family.cc:1083] [default] Increasing compaction threads because we have 4 level-0 files 2024/11/19-15:28:51.319797 139850957195008 [db_impl_compaction_flush.cc:103] [JOB 281] Syncing log #5451 2024/11/19-15:28:51.319893 139850957195008 (Original Log Time 2024/11/19-15:28:51.319589) [db_impl_compaction_flush.cc:3040] Calling FlushMemTableToOutputFile with column family [default], flush slots available 2, compaction slots available 7, flush slots scheduled 1, compaction slots scheduled 7 2024/11/19-15:28:51.319900 139850957195008 [flush_job.cc:879] [default] [JOB 281] Flushing memtable with next log file: 5458 2024/11/19-15:28:51.319928 139850957195008 EVENT_LOG_v1 {"time_micros": 1732001331319915, "job": 281, "event": "flush_started", "num_memtables": 1, "num_entries": 1316338, "num_deletes": 0, "total_data_size": 502857000, "memory_usage": 546569608, "num_range_deletes": 0, "flush_reason": "Write Buffer Full"} 2024/11/19-15:28:51.319933 139850957195008 [flush_job.cc:910] [default] [JOB 281] Level-0 flush table #5459: started 2024/11/19-15:28:51.320404 139850957195008 [cspp_memtable.cc:1074] /storage/mytopling/.rocksdb/cspp-000037.memtab-0 ConvertToReadOnly /storage/mytopling/.rocksdb/005459.sst in ConvertToSST, mem_size = 521.249 M, time = 462.756 us 2024/11/19-15:28:51.321402 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-01 2024/11/19-15:28:51.321481 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00274/att-01/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:28:51.399864 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-01 2024/11/19-15:28:51.399991 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00275/att-01/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:28:51.863501 139850957195008 [cspp_memtable.cc:1366] CSPPMemTab::ConvertToSST(/storage/mytopling/.rocksdb/005459.sst): fsize = 521.251 M, time(ms): open: 0.031, seek: 0.015, finish: 0.036, meta: 0.012, Flush: 0.011, sync: 542.965, close: 0.011, all: 543.081 2024/11/19-15:28:51.864651 139850957195008 [flush_job.cc:1083] [default] [JOB 281] Flush lasted 544757 microseconds, and 449621 cpu microseconds. 2024/11/19-15:28:51.887690 139850957195008 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 7152546 2024/11/19-15:28:51.887772 139850957195008 (Original Log Time 2024/11/19-15:28:51.863551) [flush_job.cc:1029] [default] [JOB 281] Level-0 flush table #5459: 546571034 bytes OK (needs compaction) 2024/11/19-15:28:51.887778 139850957195008 (Original Log Time 2024/11/19-15:28:51.864670) [memtable_list.cc:560] [default] Level-0 commit table #5459 started 2024/11/19-15:28:51.887781 139850957195008 (Original Log Time 2024/11/19-15:28:51.887656) [memtable_list.cc:759] [default] Level-0 commit table #5459: memtable #1 done 2024/11/19-15:28:51.887784 139850957195008 (Original Log Time 2024/11/19-15:28:51.887684) EVENT_LOG_v1 {"time_micros": 1732001331887677, "job": 281, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [5, 112, 76, 82, 0, 0, 0], "immutable_memtables": 0} 2024/11/19-15:28:51.887787 139850957195008 (Original Log Time 2024/11/19-15:28:51.887748) [db_impl_compaction_flush.cc:310] [default] Level summary: files[5 112 76 82 0 0 0] max score 5.18 (165 files need compaction) 2024/11/19-15:28:51.989648 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-01 2024/11/19-15:28:51.989722 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00277/att-01/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:28:52.211112 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-01 2024/11/19-15:28:52.211215 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00272/att-01/compact.done) = No such file or directory: 11.012 sec on http probe failed: 5, giveup 2024/11/19-15:28:52.211233 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00272/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 119.253 sec, real 11.301 (submit 283.060 ms wait 11.012 s), input 119.253 MiB, send shutdown 2024/11/19-15:28:52.219476 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-01 2024/11/19-15:28:52.219708 139851158521600 [dcompact_etcd.cc:1773] job-00272/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:52.505989 139851158521600 [dcompact_etcd.cc:1163] job-00272/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.286 sec, input = 119.253 MiB 2024/11/19-15:28:52.525749 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-01 2024/11/19-15:28:52.525813 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00278/att-01/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:28:52.569032 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-00 2024/11/19-15:28:52.569106 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00280/att-00/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:28:53.195214 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-01 2024/11/19-15:28:53.195293 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00279/att-01/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:28:53.330178 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-01 2024/11/19-15:28:53.330238 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00274/att-01/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:28:53.330256 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00274/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.907 sec, real 11.476 (submit 265.548 ms wait 11.202 s), input 126.907 MiB, send shutdown 2024/11/19-15:28:53.338942 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-01 2024/11/19-15:28:53.339165 139850978166528 [dcompact_etcd.cc:1773] job-00274/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:53.404613 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-01 2024/11/19-15:28:53.404667 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00275/att-01/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:28:53.597665 139850978166528 [dcompact_etcd.cc:1163] job-00274/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.258 sec, input = 126.907 MiB 2024/11/19-15:28:53.998772 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-01 2024/11/19-15:28:53.998847 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00277/att-01/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:28:54.530906 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-01 2024/11/19-15:28:54.530998 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00278/att-01/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:28:54.577763 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-00 2024/11/19-15:28:54.577825 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00280/att-00/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:28:55.200650 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-01 2024/11/19-15:28:55.200725 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00279/att-01/compact.done) = No such file or directory: 7.191 sec on http probe failed: 3, retry 2024/11/19-15:28:55.414160 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-01 2024/11/19-15:28:55.414228 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00275/att-01/compact.done) = No such file or directory: 9.193 sec on http probe failed: 4, retry 2024/11/19-15:28:55.493730 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-02 2024/11/19-15:28:55.493800 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00272/att-02/compact.done) = No such file or directory: 2.981 sec on http probe failed: 1, retry 2024/11/19-15:28:56.007962 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-01 2024/11/19-15:28:56.008066 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00277/att-01/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:28:56.539638 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-01 2024/11/19-15:28:56.539717 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00278/att-01/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:28:56.583581 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-00 2024/11/19-15:28:56.583643 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00280/att-00/compact.done) = No such file or directory: 11.206 sec on http probe failed: 5, giveup 2024/11/19-15:28:56.583660 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00280/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.971 sec, real 11.505 (submit 293.721 ms wait 11.206 s), input 126.971 MiB, send shutdown 2024/11/19-15:28:56.592147 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-00 2024/11/19-15:28:56.592367 139851169007360 [dcompact_etcd.cc:1773] job-00280/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:56.779011 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-02 2024/11/19-15:28:56.779122 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00274/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:28:56.883869 139851169007360 [dcompact_etcd.cc:1163] job-00280/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.291 sec, input = 126.971 MiB 2024/11/19-15:28:57.206278 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-01 2024/11/19-15:28:57.206342 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00279/att-01/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:28:57.422268 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-01 2024/11/19-15:28:57.422338 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00275/att-01/compact.done) = No such file or directory: 11.203 sec on http probe failed: 5, giveup 2024/11/19-15:28:57.422355 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00275/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.920 sec, real 11.455 (submit 244.370 ms wait 11.203 s), input 126.920 MiB, send shutdown 2024/11/19-15:28:57.427433 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-01 2024/11/19-15:28:57.427641 139851148035840 [dcompact_etcd.cc:1773] job-00275/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:57.503478 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-02 2024/11/19-15:28:57.503534 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00272/att-02/compact.done) = No such file or directory: 4.988 sec on http probe failed: 2, retry 2024/11/19-15:28:57.713022 139851148035840 [dcompact_etcd.cc:1163] job-00275/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.285 sec, input = 126.920 MiB 2024/11/19-15:28:58.013153 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-01 2024/11/19-15:28:58.013230 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00277/att-01/compact.done) = No such file or directory: 11.206 sec on http probe failed: 5, giveup 2024/11/19-15:28:58.013249 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00277/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.930 sec, real 11.489 (submit 278.433 ms wait 11.206 s), input 126.930 MiB, send shutdown 2024/11/19-15:28:58.021963 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-01 2024/11/19-15:28:58.022182 139851179493120 [dcompact_etcd.cc:1773] job-00277/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:58.305444 139851179493120 [dcompact_etcd.cc:1163] job-00277/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.283 sec, input = 126.930 MiB 2024/11/19-15:28:58.547311 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-01 2024/11/19-15:28:58.547387 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00278/att-01/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:28:58.547406 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00278/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.908 sec, real 11.503 (submit 291.084 ms wait 11.204 s), input 126.908 MiB, send shutdown 2024/11/19-15:28:58.555945 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-01 2024/11/19-15:28:58.556194 139851137550080 [dcompact_etcd.cc:1773] job-00278/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:58.792010 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-02 2024/11/19-15:28:58.792104 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00274/att-02/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:28:58.834592 139851137550080 [dcompact_etcd.cc:1163] job-00278/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.278 sec, input = 126.908 MiB 2024/11/19-15:28:59.215904 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-01 2024/11/19-15:28:59.216003 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00279/att-01/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:28:59.216022 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00279/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.846 sec, real 11.522 (submit 310.016 ms wait 11.202 s), input 126.846 MiB, send shutdown 2024/11/19-15:28:59.223860 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-01 2024/11/19-15:28:59.224081 139851127064320 [dcompact_etcd.cc:1773] job-00279/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:28:59.502148 139851127064320 [dcompact_etcd.cc:1163] job-00279/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.278 sec, input = 126.846 MiB 2024/11/19-15:28:59.509573 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-02 2024/11/19-15:28:59.509631 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00272/att-02/compact.done) = No such file or directory: 6.998 sec on http probe failed: 3, retry 2024/11/19-15:29:00.064425 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-01 2024/11/19-15:29:00.064513 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00280/att-01/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:29:00.801708 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-02 2024/11/19-15:29:00.801803 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00274/att-02/compact.done) = No such file or directory: 7.195 sec on http probe failed: 3, retry 2024/11/19-15:29:00.896491 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-02 2024/11/19-15:29:00.896600 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00275/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:01.487782 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-02 2024/11/19-15:29:01.487870 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00277/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:01.518513 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-02 2024/11/19-15:29:01.518607 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00272/att-02/compact.done) = No such file or directory: 9.004 sec on http probe failed: 4, retry 2024/11/19-15:29:02.016658 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-02 2024/11/19-15:29:02.016752 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00278/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:02.072972 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-01 2024/11/19-15:29:02.073076 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00280/att-01/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:29:02.682517 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-02 2024/11/19-15:29:02.682601 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00279/att-02/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:29:02.810902 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-02 2024/11/19-15:29:02.810980 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00274/att-02/compact.done) = No such file or directory: 9.204 sec on http probe failed: 4, retry 2024/11/19-15:29:02.905362 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-02 2024/11/19-15:29:02.905429 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00275/att-02/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:29:03.493828 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-02 2024/11/19-15:29:03.493909 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00277/att-02/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:29:03.528347 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-02 2024/11/19-15:29:03.528420 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00272/att-02/compact.done) = No such file or directory: 11.013 sec on http probe failed: 5, giveup 2024/11/19-15:29:03.528439 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00272/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 119.253 sec, real 11.309 (submit 286.205 ms wait 11.013 s), input 119.253 MiB, send shutdown 2024/11/19-15:29:03.537274 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00272/att-02 2024/11/19-15:29:03.537492 139851158521600 [dcompact_etcd.cc:1773] job-00272/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:03.539352 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00272) = success 2024/11/19-15:29:03.539366 139851158521600 [dcompact_etcd.cc:1773] job-00272/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:29:03.539386 139851158521600 (Original Log Time 2024/11/19-15:28:29.599076) EVENT_LOG_v1 {"time_micros": 1732001309599069, "job": 272, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5380, 5381, 5252, 5255], "score": 6.91206, "input_data_size": 126146368, "oldest_snapshot_seqno": 156233682} 2024/11/19-15:29:03.539393 139851158521600 [compaction_job.cc:2437] [default] [JOB 272] Compacting 4@1 files to L2, score 6.91, subcompactions 1 : 1 2024/11/19-15:29:03.539401 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 270 Base level 1, inputs: [5380(31MB) 5381(24MB) 5252(31MB) 5255(31MB)] 2024/11/19-15:29:03.539609 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005460.sst) /storage/mytopling/..//tmp/Topling-82038-Jkk0aP 2024/11/19-15:29:03.592545 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005460.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:29:03.595839 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005460.sst): index pass time = 0.00's, 158.725'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64716 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:29:03.658652 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005460.sst): first pass time = 0.12's, 563.803'MB/sec 2024/11/19-15:29:03.658708 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005460.sst): old prealloc_size = 73819750, real_size = 66151664 2024/11/19-15:29:03.658767 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005460.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64716 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.98 avg-zval =1020.23 2024/11/19-15:29:03.658811 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005460.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.067 GB 2024/11/19-15:29:03.718450 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005460.sst): second pass time = 0.00's,159744101.942'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1107.360'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2024.915'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.378'MB/sec write SST data time = 0.02's, 3920.913'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 159744101.942'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64716 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 1020.23 usrkeys = 64716 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 1020.23 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0466 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 19.948683 GB avg = 0.262 KB (by entry num) total key len = 1.966010 GB avg = 0.026 KB total ukey len = 1.356585 GB avg = 0.018 KB total ukey num = 0.076178159 Billion total entry num = 0.076178159 Billion write speed all = 34.965589906 MB/sec (with seq num) write speed all = 33.993232504 MB/sec (without seq num) 2024/11/19-15:29:03.722082 139851158521600 [compaction_job.cc:1948] [default] [JOB 272] Generated table #5460: 64716 keys, 66136900 bytes (need compaction), temperature: kUnknown 2024/11/19-15:29:03.722135 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001343722112, "cf_name": "default", "job": 272, "event": "table_file_creation", "file_number": 5460, "file_size": 66136900, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025120, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294320, "raw_average_key_size": 20, "raw_value_size": 65814570, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64716, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001343, "oldest_key_time": 0, "file_creation_time": 1732001343, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5460, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:29:03.736531 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005461.sst) /storage/mytopling/..//tmp/Topling-82038-SVUv4T 2024/11/19-15:29:03.781578 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005461.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.058 GB 2024/11/19-15:29:03.781691 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005461.sst): index pass time = 0.00's,5429.477'MB/sec index type = UintIndex_AllOne usrkeys = 55870 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0004 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:29:03.838896 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005461.sst): first pass time = 0.10's, 566.079'MB/sec 2024/11/19-15:29:03.838954 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005461.sst): old prealloc_size = 73819750, real_size = 57015240 2024/11/19-15:29:03.838977 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005461.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 55870 prefix = 4 raw-key = 0.0004 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0568 GB zip-val = 0.0570 GB avg-val =1016.99 avg-zval =1020.24 2024/11/19-15:29:03.885593 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005461.sst): second pass time = 0.00's,207369010.949'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1222.637'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.636'MB/sec (index lex order gen) rebuild zvType time = 0.03's, 0.422'MB/sec write SST data time = 0.01's, 4225.864'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.057'GB zip my value throughput = 207369010.949'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 55870 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.99 avg-zval = 1020.24 usrkeys = 55870 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.99 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0007 GB value = 0.0568 GB dict = 0.00 MB all = 0.0579 GB } __ZipSize{ index = 0.0000 GB value = 0.0570 GB dict = 0.00 MB all = 0.0570 GB } UnZip/Zip{ index =3491.8750 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 20.005502 GB avg = 0.262 KB (by entry num) total key len = 1.967128 GB avg = 0.026 KB total ukey len = 1.357256 GB avg = 0.018 KB total ukey num = 0.076234029 Billion total entry num = 0.076234029 Billion write speed all = 35.048682597 MB/sec (with seq num) write speed all = 34.075871487 MB/sec (without seq num) 2024/11/19-15:29:03.888937 139851158521600 [compaction_job.cc:1948] [default] [JOB 272] Generated table #5461: 55870 keys, 57002250 bytes (need compaction), temperature: kUnknown 2024/11/19-15:29:03.888991 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001343888967, "cf_name": "default", "job": 272, "event": "table_file_creation", "file_number": 5461, "file_size": 57002250, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 57000912, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1117400, "raw_average_key_size": 20, "raw_value_size": 56819109, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 55870, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001343, "oldest_key_time": 0, "file_creation_time": 1732001343, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5461, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:29:03.901376 139851158521600 [compaction_job.cc:703] job-00272: subcompact[0], size: 0.123139 G, files: /storage/mytopling/.rocksdb [005460,005461] 2024/11/19-15:29:03.902164 139851158521600 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 7373758 2024/11/19-15:29:03.902496 139851158521600 (Original Log Time 2024/11/19-15:29:03.539429) EVENT_LOG_v1 {"time_micros": 1732001343539414, "job": 272, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5380, 5381, 5252, 5255], "score": 6.91206, "input_data_size": 126146368, "oldest_snapshot_seqno": 156233682} 2024/11/19-15:29:03.902503 139851158521600 (Original Log Time 2024/11/19-15:29:03.901758) [compaction_job.cc:2022] [default] [JOB 272] Compacted 4@1 files to L2 => 123139150 bytes 2024/11/19-15:29:03.902507 139851158521600 (Original Log Time 2024/11/19-15:29:03.902132) [compaction_job.cc:1224] [default] compacted to: files[5 108 78 82 0 0 0] max score 5.18 (167 files need compaction), MB/sec: 348.5 rd, 340.2 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(120.3, 0.0 +0.0 blob) out(117.4 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 120586, records dropped: 0 output_compression: Snappy 2024/11/19-15:29:03.902526 139851158521600 (Original Log Time 2024/11/19-15:29:03.902157) EVENT_LOG_v1 {"time_micros": 1732001343902144, "cf": "default", "job": 272, "event": "compaction_finished", "compaction_time_micros": 361960, "compaction_time_cpu_micros": 360465, "output_level": 2, "num_output_files": 2, "total_output_size": 123139150, "num_input_records": 120586, "num_output_records": 120586, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 108, 78, 82, 0, 0, 0]} 2024/11/19-15:29:03.906411 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005307.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:03.906435 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001343906432, "job": 272, "event": "table_file_deletion", "file_number": 5307} 2024/11/19-15:29:03.910175 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005312.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:03.910196 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001343910194, "job": 272, "event": "table_file_deletion", "file_number": 5312} 2024/11/19-15:29:03.914031 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005316.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:03.914051 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001343914049, "job": 272, "event": "table_file_deletion", "file_number": 5316} 2024/11/19-15:29:03.917947 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005318.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:03.917965 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001343917963, "job": 272, "event": "table_file_deletion", "file_number": 5318} 2024/11/19-15:29:03.918091 139851158521600 [compaction_job.cc:2437] [default] [JOB 282] Compacting 4@1 files to L2, score 5.18, subcompactions 1 : 1 2024/11/19-15:29:03.918099 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 280 Base level 1, inputs: [5302(31MB) 5304(31MB) 5306(31MB) 5308(31MB)] 2024/11/19-15:29:04.024228 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-02 2024/11/19-15:29:04.024305 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00278/att-02/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:29:04.082080 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-01 2024/11/19-15:29:04.082143 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00280/att-01/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:29:04.219576 139851158521600 [dcompact_etcd.cc:1163] job-00282/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.301 sec, input = 126.808 MiB 2024/11/19-15:29:04.692205 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-02 2024/11/19-15:29:04.692279 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00279/att-02/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:29:04.815751 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-02 2024/11/19-15:29:04.815810 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00274/att-02/compact.done) = No such file or directory: 11.213 sec on http probe failed: 5, giveup 2024/11/19-15:29:04.815828 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00274/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.907 sec, real 11.477 (submit 258.427 ms wait 11.213 s), input 126.907 MiB, send shutdown 2024/11/19-15:29:04.820410 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00274/att-02 2024/11/19-15:29:04.820619 139850978166528 [dcompact_etcd.cc:1773] job-00274/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:04.823381 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00274) = success 2024/11/19-15:29:04.823388 139850978166528 [dcompact_etcd.cc:1773] job-00274/att-03: CleanFiles 0.003 sec : 0.000 2024/11/19-15:29:04.823407 139850978166528 (Original Log Time 2024/11/19-15:28:30.333089) EVENT_LOG_v1 {"time_micros": 1732001310333084, "job": 274, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5257, 5259, 5262, 5263], "score": 6.67709, "input_data_size": 134099648, "oldest_snapshot_seqno": 156281970} 2024/11/19-15:29:04.823414 139850978166528 [compaction_job.cc:2437] [default] [JOB 274] Compacting 4@1 files to L2, score 6.68, subcompactions 1 : 1 2024/11/19-15:29:04.823422 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 271 Base level 1, inputs: [5257(31MB) 5259(31MB) 5262(31MB) 5263(31MB)] 2024/11/19-15:29:04.823644 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005462.sst) /storage/mytopling/..//tmp/Topling-82038-Cmeo8r 2024/11/19-15:29:04.876813 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005462.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:29:04.880147 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005462.sst): index pass time = 0.00's, 156.900'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64714 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:29:04.915185 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-02 2024/11/19-15:29:04.915263 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00275/att-02/compact.done) = No such file or directory: 7.192 sec on http probe failed: 3, retry 2024/11/19-15:29:04.943141 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005462.sst): first pass time = 0.12's, 561.650'MB/sec 2024/11/19-15:29:04.943188 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005462.sst): old prealloc_size = 73819750, real_size = 66151728 2024/11/19-15:29:04.943244 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005462.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64714 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.01 avg-zval =1020.26 2024/11/19-15:29:04.943296 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005462.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.067 GB 2024/11/19-15:29:05.002879 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005462.sst): second pass time = 0.00's,234215829.181'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1108.597'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2129.623'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.380'MB/sec write SST data time = 0.02's, 3890.020'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 234215829.181'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64714 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 usrkeys = 64714 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0464 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 20.071316 GB avg = 0.263 KB (by entry num) total key len = 1.968422 GB avg = 0.026 KB total ukey len = 1.358032 GB avg = 0.018 KB total ukey num = 0.076298743 Billion total entry num = 0.076298743 Billion write speed all = 35.093186540 MB/sec (with seq num) write speed all = 34.121281722 MB/sec (without seq num) 2024/11/19-15:29:05.006488 139850978166528 [compaction_job.cc:1948] [default] [JOB 274] Generated table #5462: 64714 keys, 66136964 bytes (need compaction), temperature: kUnknown 2024/11/19-15:29:05.006541 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001345006517, "cf_name": "default", "job": 274, "event": "table_file_creation", "file_number": 5462, "file_size": 66136964, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025184, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294280, "raw_average_key_size": 20, "raw_value_size": 65814648, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 64714, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001344, "oldest_key_time": 0, "file_creation_time": 1732001344, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5462, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:29:05.020962 139850978166528 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005463.sst) /storage/mytopling/..//tmp/Topling-82038-BrqYhk 2024/11/19-15:29:05.072592 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005463.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:29:05.075892 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005463.sst): index pass time = 0.00's, 155.827'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63613 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:29:05.138707 139850978166528 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005463.sst): first pass time = 0.12's, 560.303'MB/sec 2024/11/19-15:29:05.138762 139850978166528 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005463.sst): old prealloc_size = 73819750, real_size = 65022336 2024/11/19-15:29:05.138818 139850978166528 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005463.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63613 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1016.95 avg-zval =1020.20 2024/11/19-15:29:05.138884 139850978166528 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005463.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.066 GB 2024/11/19-15:29:05.197961 139850978166528 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005463.sst): second pass time = 0.00's,261906615.385'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1098.542'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2106.950'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.376'MB/sec write SST data time = 0.02's, 3855.514'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 261906615.385'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63613 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.95 avg-zval = 1020.20 usrkeys = 63613 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.95 avg-zval = 1020.20 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7927 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0451 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 20.136007 GB avg = 0.264 KB (by entry num) total key len = 1.969694 GB avg = 0.026 KB total ukey len = 1.358796 GB avg = 0.018 KB total ukey num = 0.076362356 Billion total entry num = 0.076362356 Billion write speed all = 35.187287276 MB/sec (with seq num) write speed all = 34.214874211 MB/sec (without seq num) 2024/11/19-15:29:05.201610 139850978166528 [compaction_job.cc:1948] [default] [JOB 274] Generated table #5463: 63613 keys, 65007812 bytes (need compaction), temperature: kUnknown 2024/11/19-15:29:05.201662 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001345201638, "cf_name": "default", "job": 274, "event": "table_file_creation", "file_number": 5463, "file_size": 65007812, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64897904, "index_size": 108352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1272260, "raw_average_key_size": 20, "raw_value_size": 64690934, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63613, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001345, "oldest_key_time": 0, "file_creation_time": 1732001345, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5463, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:29:05.215809 139850978166528 [compaction_job.cc:703] job-00274: subcompact[0], size: 0.131145 G, files: /storage/mytopling/.rocksdb [005462,005463] 2024/11/19-15:29:05.216637 139850978166528 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 7601812 2024/11/19-15:29:05.216797 139850978166528 (Original Log Time 2024/11/19-15:29:04.823449) EVENT_LOG_v1 {"time_micros": 1732001344823434, "job": 274, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5257, 5259, 5262, 5263], "score": 6.67709, "input_data_size": 134099648, "oldest_snapshot_seqno": 156281970} 2024/11/19-15:29:05.216807 139850978166528 (Original Log Time 2024/11/19-15:29:05.216255) [compaction_job.cc:2022] [default] [JOB 274] Compacted 4@1 files to L2 => 131144776 bytes 2024/11/19-15:29:05.216813 139850978166528 (Original Log Time 2024/11/19-15:29:05.216605) [compaction_job.cc:1224] [default] compacted to: files[5 104 80 82 0 0 0] max score 4.93 (165 files need compaction), MB/sec: 341.8 rd, 334.2 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.1 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128327, records dropped: 0 output_compression: Snappy 2024/11/19-15:29:05.216837 139850978166528 (Original Log Time 2024/11/19-15:29:05.216630) EVENT_LOG_v1 {"time_micros": 1732001345216617, "cf": "default", "job": 274, "event": "compaction_finished", "compaction_time_micros": 392373, "compaction_time_cpu_micros": 390673, "output_level": 2, "num_output_files": 2, "total_output_size": 131144776, "num_input_records": 128327, "num_output_records": 128327, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 104, 80, 82, 0, 0, 0]} 2024/11/19-15:29:05.219864 139850978166528 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005377.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:05.219890 139850978166528 EVENT_LOG_v1 {"time_micros": 1732001345219887, "job": 274, "event": "table_file_deletion", "file_number": 5377} 2024/11/19-15:29:05.220134 139850978166528 [compaction_job.cc:2437] [default] [JOB 283] Compacting 4@1 files to L2, score 4.93, subcompactions 1 : 1 2024/11/19-15:29:05.220144 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 281 Base level 1, inputs: [5309(31MB) 5310(31MB) 5311(31MB) 5313(31MB)] 2024/11/19-15:29:05.499821 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-02 2024/11/19-15:29:05.499907 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00277/att-02/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:29:05.500954 139850978166528 [dcompact_etcd.cc:1163] job-00283/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 126.914 MiB 2024/11/19-15:29:06.029210 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-02 2024/11/19-15:29:06.029282 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00278/att-02/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:29:06.091408 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-01 2024/11/19-15:29:06.091469 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00280/att-01/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:29:06.697158 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-02 2024/11/19-15:29:06.697251 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00279/att-02/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:29:06.924140 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-02 2024/11/19-15:29:06.924237 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00275/att-02/compact.done) = No such file or directory: 9.202 sec on http probe failed: 4, retry 2024/11/19-15:29:07.399276 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00282/att-00 2024/11/19-15:29:07.399363 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00282/att-00/compact.done) = No such file or directory: 3.170 sec on http probe failed: 1, retry 2024/11/19-15:29:07.509749 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-02 2024/11/19-15:29:07.509818 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00277/att-02/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:29:08.034069 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-02 2024/11/19-15:29:08.034143 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00278/att-02/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:29:08.100338 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-01 2024/11/19-15:29:08.100396 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00280/att-01/compact.done) = No such file or directory: 11.208 sec on http probe failed: 5, giveup 2024/11/19-15:29:08.100414 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00280/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.971 sec, real 11.508 (submit 291.426 ms wait 11.208 s), input 126.971 MiB, send shutdown 2024/11/19-15:29:08.108465 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-01 2024/11/19-15:29:08.108684 139851169007360 [dcompact_etcd.cc:1773] job-00280/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:08.393239 139851169007360 [dcompact_etcd.cc:1163] job-00280/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.284 sec, input = 126.971 MiB 2024/11/19-15:29:08.681063 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00283/att-00 2024/11/19-15:29:08.681143 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00283/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:08.705674 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-02 2024/11/19-15:29:08.705733 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00279/att-02/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:29:08.934031 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-02 2024/11/19-15:29:08.934108 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00275/att-02/compact.done) = No such file or directory: 11.211 sec on http probe failed: 5, giveup 2024/11/19-15:29:08.934127 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00275/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.920 sec, real 11.506 (submit 285.310 ms wait 11.211 s), input 126.920 MiB, send shutdown 2024/11/19-15:29:08.943230 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00275/att-02 2024/11/19-15:29:08.943441 139851148035840 [dcompact_etcd.cc:1773] job-00275/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:08.945752 139851148035840 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00275) = success 2024/11/19-15:29:08.945761 139851148035840 [dcompact_etcd.cc:1773] job-00275/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:29:08.945779 139851148035840 (Original Log Time 2024/11/19-15:28:34.452431) EVENT_LOG_v1 {"time_micros": 1732001314452425, "job": 275, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5265, 5268, 5270, 5272], "score": 6.42731, "input_data_size": 134098880, "oldest_snapshot_seqno": 156622931} 2024/11/19-15:29:08.945793 139851148035840 [compaction_job.cc:2437] [default] [JOB 275] Compacting 4@1 files to L2, score 6.43, subcompactions 1 : 1 2024/11/19-15:29:08.945801 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 273 Base level 1, inputs: [5265(31MB) 5268(31MB) 5270(31MB) 5272(31MB)] 2024/11/19-15:29:08.946038 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005464.sst) /storage/mytopling/..//tmp/Topling-82038-ZWJCWl 2024/11/19-15:29:08.999432 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005464.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:29:08.999540 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005464.sst): index pass time = 0.00's,6602.494'MB/sec index type = UintIndex_AllOne usrkeys = 64716 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 6 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:29:09.066042 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005464.sst): first pass time = 0.12's, 559.288'MB/sec 2024/11/19-15:29:09.066093 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005464.sst): old prealloc_size = 73819750, real_size = 66042296 2024/11/19-15:29:09.066116 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005464.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64716 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.99 avg-zval =1020.24 2024/11/19-15:29:09.120825 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005464.sst): second pass time = 0.00's,236745496.403'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1206.697'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.337'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.415'MB/sec write SST data time = 0.02's, 4205.571'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 236745496.403'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64716 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.99 avg-zval = 1020.24 usrkeys = 64716 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.99 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 6 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.7500 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 20.201823 GB avg = 0.264 KB (by entry num) total key len = 1.970989 GB avg = 0.026 KB total ukey len = 1.359572 GB avg = 0.018 KB total ukey num = 0.076427072 Billion total entry num = 0.076427072 Billion write speed all = 35.075090206 MB/sec (with seq num) write speed all = 34.107892534 MB/sec (without seq num) 2024/11/19-15:29:09.124529 139851148035840 [compaction_job.cc:1948] [default] [JOB 275] Generated table #5464: 64716 keys, 66027082 bytes (need compaction), temperature: kUnknown 2024/11/19-15:29:09.124579 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001349124556, "cf_name": "default", "job": 275, "event": "table_file_creation", "file_number": 5464, "file_size": 66027082, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025792, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294320, "raw_average_key_size": 20, "raw_value_size": 65815248, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64716, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001348, "oldest_key_time": 0, "file_creation_time": 1732001348, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5464, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:29:09.141339 139851148035840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005465.sst) /storage/mytopling/..//tmp/Topling-82038-2sUgX3 2024/11/19-15:29:09.193648 139834035275520 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005465.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:29:09.198900 139834035275520 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005465.sst): index pass time = 0.01's, 97.648'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63623 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:29:09.260629 139851148035840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005465.sst): first pass time = 0.12's, 553.162'MB/sec 2024/11/19-15:29:09.260683 139851148035840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005465.sst): old prealloc_size = 73819750, real_size = 65034864 2024/11/19-15:29:09.260740 139851148035840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005465.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63623 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1016.98 avg-zval =1020.23 2024/11/19-15:29:09.260803 139851148035840 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005465.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.066 GB 2024/11/19-15:29:09.319638 139851148035840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005465.sst): second pass time = 0.00's,300945930.233'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1103.150'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2044.582'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.378'MB/sec write SST data time = 0.02's, 3856.833'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 300945930.233'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63623 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 1020.23 usrkeys = 63623 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 1020.23 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7928 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0421 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 20.266526 GB avg = 0.265 KB (by entry num) total key len = 1.972261 GB avg = 0.026 KB total ukey len = 1.360336 GB avg = 0.018 KB total ukey num = 0.076490695 Billion total entry num = 0.076490695 Billion write speed all = 35.168396616 MB/sec (with seq num) write speed all = 34.200698126 MB/sec (without seq num) 2024/11/19-15:29:09.323233 139851148035840 [compaction_job.cc:1948] [default] [JOB 275] Generated table #5465: 63623 keys, 65020356 bytes (need compaction), temperature: kUnknown 2024/11/19-15:29:09.323285 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001349323261, "cf_name": "default", "job": 275, "event": "table_file_creation", "file_number": 5465, "file_size": 65020356, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64910368, "index_size": 108416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1272460, "raw_average_key_size": 20, "raw_value_size": 64703375, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63623, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001349, "oldest_key_time": 0, "file_creation_time": 1732001349, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5465, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:29:09.337316 139851148035840 [compaction_job.cc:703] job-00275: subcompact[0], size: 0.131047 G, files: /storage/mytopling/.rocksdb [005464,005465] 2024/11/19-15:29:09.338103 139851148035840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 7836919 2024/11/19-15:29:09.338245 139851148035840 (Original Log Time 2024/11/19-15:29:08.945831) EVENT_LOG_v1 {"time_micros": 1732001348945814, "job": 275, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5265, 5268, 5270, 5272], "score": 6.42731, "input_data_size": 134098880, "oldest_snapshot_seqno": 156622931} 2024/11/19-15:29:09.338252 139851148035840 (Original Log Time 2024/11/19-15:29:09.337701) [compaction_job.cc:2022] [default] [JOB 275] Compacted 4@1 files to L2 => 131047438 bytes 2024/11/19-15:29:09.338255 139851148035840 (Original Log Time 2024/11/19-15:29:09.338072) [compaction_job.cc:1224] [default] compacted to: files[5 100 82 82 0 0 0] max score 4.68 (163 files need compaction), MB/sec: 342.5 rd, 334.7 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128339, records dropped: 0 output_compression: Snappy 2024/11/19-15:29:09.338276 139851148035840 (Original Log Time 2024/11/19-15:29:09.338096) EVENT_LOG_v1 {"time_micros": 1732001349338085, "cf": "default", "job": 275, "event": "compaction_finished", "compaction_time_micros": 391498, "compaction_time_cpu_micros": 389868, "output_level": 2, "num_output_files": 2, "total_output_size": 131047438, "num_input_records": 128339, "num_output_records": 128339, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 100, 82, 82, 0, 0, 0]} 2024/11/19-15:29:09.342049 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005323.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:09.342073 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001349342070, "job": 275, "event": "table_file_deletion", "file_number": 5323} 2024/11/19-15:29:09.346131 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005326.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:09.346152 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001349346150, "job": 275, "event": "table_file_deletion", "file_number": 5326} 2024/11/19-15:29:09.350362 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005329.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:09.350382 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001349350380, "job": 275, "event": "table_file_deletion", "file_number": 5329} 2024/11/19-15:29:09.354990 139851148035840 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005333.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:09.355010 139851148035840 EVENT_LOG_v1 {"time_micros": 1732001349355008, "job": 275, "event": "table_file_deletion", "file_number": 5333} 2024/11/19-15:29:09.355143 139851148035840 [compaction_job.cc:2437] [default] [JOB 284] Compacting 4@1 files to L2, score 4.68, subcompactions 1 : 1 2024/11/19-15:29:09.355152 139851148035840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 282 Base level 1, inputs: [5314(31MB) 5315(31MB) 5317(31MB) 5319(31MB)] 2024/11/19-15:29:09.407540 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00282/att-00 2024/11/19-15:29:09.407602 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00282/att-00/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:29:09.519420 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-02 2024/11/19-15:29:09.519473 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00277/att-02/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:29:09.519489 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00277/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.930 sec, real 11.497 (submit 283.185 ms wait 11.204 s), input 126.930 MiB, send shutdown 2024/11/19-15:29:09.528158 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00277/att-02 2024/11/19-15:29:09.528351 139851179493120 [dcompact_etcd.cc:1773] job-00277/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:09.530869 139851179493120 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00277) = success 2024/11/19-15:29:09.530877 139851179493120 [dcompact_etcd.cc:1773] job-00277/att-03: CleanFiles 0.003 sec : 0.000 2024/11/19-15:29:09.530893 139851179493120 (Original Log Time 2024/11/19-15:28:34.991849) EVENT_LOG_v1 {"time_micros": 1732001314991835, "job": 277, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5275, 5277, 5278, 5281], "score": 6.17753, "input_data_size": 134099264, "oldest_snapshot_seqno": -1} 2024/11/19-15:29:09.530899 139851179493120 [compaction_job.cc:2437] [default] [JOB 277] Compacting 4@1 files to L2, score 6.18, subcompactions 1 : 1 2024/11/19-15:29:09.530907 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 274 Base level 1, inputs: [5275(31MB) 5277(31MB) 5278(31MB) 5281(31MB)] 2024/11/19-15:29:09.531128 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005466.sst) /storage/mytopling/..//tmp/Topling-82038-gS71Tf 2024/11/19-15:29:09.583540 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005466.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:29:09.583651 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005466.sst): index pass time = 0.00's,6227.087'MB/sec index type = UintIndex_AllOne usrkeys = 64715 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:29:09.649785 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005466.sst): first pass time = 0.12's, 565.634'MB/sec 2024/11/19-15:29:09.649836 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005466.sst): old prealloc_size = 73819750, real_size = 66041944 2024/11/19-15:29:09.649858 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005466.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64715 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.00 avg-zval =1020.25 2024/11/19-15:29:09.656931 139851148035840 [dcompact_etcd.cc:1163] job-00284/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.301 sec, input = 126.909 MiB 2024/11/19-15:29:09.704078 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005466.sst): second pass time = 0.00's,378246591.954'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1217.638'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.276'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.417'MB/sec write SST data time = 0.02's, 4273.132'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 378246591.954'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64715 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.00 avg-zval = 1020.25 usrkeys = 64715 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.00 avg-zval = 1020.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.6875 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 20.332341 GB avg = 0.266 KB (by entry num) total key len = 1.973555 GB avg = 0.026 KB total ukey len = 1.361112 GB avg = 0.018 KB total ukey num = 0.076555410 Billion total entry num = 0.076555410 Billion write speed all = 35.253090858 MB/sec (with seq num) write speed all = 34.285162102 MB/sec (without seq num) 2024/11/19-15:29:09.707783 139851179493120 [compaction_job.cc:1948] [default] [JOB 277] Generated table #5466: 64715 keys, 66026762 bytes (need compaction), temperature: kUnknown 2024/11/19-15:29:09.707835 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001349707813, "cf_name": "default", "job": 277, "event": "table_file_creation", "file_number": 5466, "file_size": 66026762, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025440, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294300, "raw_average_key_size": 20, "raw_value_size": 65814907, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64715, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001349, "oldest_key_time": 0, "file_creation_time": 1732001349, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5466, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:29:09.722229 139851179493120 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005467.sst) /storage/mytopling/..//tmp/Topling-82038-PuhVmf 2024/11/19-15:29:09.773183 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005467.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:29:09.776456 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005467.sst): index pass time = 0.00's, 157.098'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63633 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:29:09.838504 139851179493120 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005467.sst): first pass time = 0.12's, 567.602'MB/sec 2024/11/19-15:29:09.838552 139851179493120 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005467.sst): old prealloc_size = 73819750, real_size = 65045728 2024/11/19-15:29:09.838610 139851179493120 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005467.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63633 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1016.99 avg-zval =1020.24 2024/11/19-15:29:09.838664 139851179493120 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005467.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.065 sec, Key+Value = 0.066 GB 2024/11/19-15:29:09.897785 139851179493120 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005467.sst): second pass time = 0.00's,273055704.641'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1098.388'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2024.915'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.375'MB/sec write SST data time = 0.02's, 3876.169'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 273055704.641'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63633 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 usrkeys = 63633 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.99 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7929 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0649 GB dict = 0.00 MB all = 0.0650 GB } UnZip/Zip{ index = 7.0432 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 20.397055 GB avg = 0.266 KB (by entry num) total key len = 1.974828 GB avg = 0.026 KB total ukey len = 1.361876 GB avg = 0.018 KB total ukey num = 0.076619043 Billion total entry num = 0.076619043 Billion write speed all = 35.346558460 MB/sec (with seq num) write speed all = 34.378121625 MB/sec (without seq num) 2024/11/19-15:29:09.901420 139851179493120 [compaction_job.cc:1948] [default] [JOB 277] Generated table #5467: 63633 keys, 65031236 bytes (need compaction), temperature: kUnknown 2024/11/19-15:29:09.901470 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001349901447, "cf_name": "default", "job": 277, "event": "table_file_creation", "file_number": 5467, "file_size": 65031236, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64921232, "index_size": 108416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1272660, "raw_average_key_size": 20, "raw_value_size": 64714202, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63633, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001349, "oldest_key_time": 0, "file_creation_time": 1732001349, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5467, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:29:09.915558 139851179493120 [compaction_job.cc:703] job-00277: subcompact[0], size: 0.131058 G, files: /storage/mytopling/.rocksdb [005466,005467] 2024/11/19-15:29:09.916321 139851179493120 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 8079297 2024/11/19-15:29:09.916481 139851179493120 (Original Log Time 2024/11/19-15:29:09.530937) EVENT_LOG_v1 {"time_micros": 1732001349530916, "job": 277, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5275, 5277, 5278, 5281], "score": 6.17753, "input_data_size": 134099264, "oldest_snapshot_seqno": -1} 2024/11/19-15:29:09.916492 139851179493120 (Original Log Time 2024/11/19-15:29:09.915952) [compaction_job.cc:2022] [default] [JOB 277] Compacted 4@1 files to L2 => 131057998 bytes 2024/11/19-15:29:09.916498 139851179493120 (Original Log Time 2024/11/19-15:29:09.916293) [compaction_job.cc:1224] [default] compacted to: files[5 96 84 82 0 0 0] max score 4.43 (161 files need compaction), MB/sec: 348.6 rd, 340.7 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128348, records dropped: 0 output_compression: Snappy 2024/11/19-15:29:09.916523 139851179493120 (Original Log Time 2024/11/19-15:29:09.916315) EVENT_LOG_v1 {"time_micros": 1732001349916303, "cf": "default", "job": 277, "event": "compaction_finished", "compaction_time_micros": 384634, "compaction_time_cpu_micros": 383012, "output_level": 2, "num_output_files": 2, "total_output_size": 131057998, "num_input_records": 128348, "num_output_records": 128348, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 96, 84, 82, 0, 0, 0]} 2024/11/19-15:29:09.920442 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005336.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:09.920467 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001349920464, "job": 277, "event": "table_file_deletion", "file_number": 5336} 2024/11/19-15:29:09.924446 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005339.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:09.924468 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001349924466, "job": 277, "event": "table_file_deletion", "file_number": 5339} 2024/11/19-15:29:09.928442 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005342.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:09.928462 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001349928460, "job": 277, "event": "table_file_deletion", "file_number": 5342} 2024/11/19-15:29:09.932660 139851179493120 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005344.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:09.932679 139851179493120 EVENT_LOG_v1 {"time_micros": 1732001349932677, "job": 277, "event": "table_file_deletion", "file_number": 5344} 2024/11/19-15:29:09.932834 139851179493120 [compaction_job.cc:2437] [default] [JOB 285] Compacting 4@1 files to L2, score 4.43, subcompactions 1 : 1 2024/11/19-15:29:09.932843 139851179493120 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 283 Base level 1, inputs: [5320(31MB) 5321(31MB) 5322(31MB) 5324(31MB)] 2024/11/19-15:29:10.039398 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-02 2024/11/19-15:29:10.039501 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00278/att-02/compact.done) = No such file or directory: 11.200 sec on http probe failed: 5, giveup 2024/11/19-15:29:10.039527 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00278/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.908 sec, real 11.483 (submit 278.316 ms wait 11.200 s), input 126.908 MiB, send shutdown 2024/11/19-15:29:10.048281 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00278/att-02 2024/11/19-15:29:10.048538 139851137550080 [dcompact_etcd.cc:1773] job-00278/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:10.050434 139851137550080 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00278) = success 2024/11/19-15:29:10.050450 139851137550080 [dcompact_etcd.cc:1773] job-00278/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:29:10.050470 139851137550080 (Original Log Time 2024/11/19-15:28:35.561523) EVENT_LOG_v1 {"time_micros": 1732001315561516, "job": 278, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5283, 5285, 5287, 5290], "score": 5.92775, "input_data_size": 134098752, "oldest_snapshot_seqno": 156682599} 2024/11/19-15:29:10.050477 139851137550080 [compaction_job.cc:2437] [default] [JOB 278] Compacting 4@1 files to L2, score 5.93, subcompactions 1 : 1 2024/11/19-15:29:10.050485 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 276 Base level 1, inputs: [5283(31MB) 5285(31MB) 5287(31MB) 5290(31MB)] 2024/11/19-15:29:10.050719 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005468.sst) /storage/mytopling/..//tmp/Topling-82038-3pviXK 2024/11/19-15:29:10.104530 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005468.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:29:10.107894 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005468.sst): index pass time = 0.00's, 155.394'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64716 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:29:10.171552 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005468.sst): first pass time = 0.12's, 555.453'MB/sec 2024/11/19-15:29:10.171605 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005468.sst): old prealloc_size = 73819750, real_size = 66152160 2024/11/19-15:29:10.171662 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005468.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64716 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.98 avg-zval =1020.24 2024/11/19-15:29:10.172826 139851137550080 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005468.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.068 sec, Key+Value = 0.067 GB 2024/11/19-15:29:10.239559 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005468.sst): second pass time = 0.00's,129049160.784'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 973.507'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2078.926'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.375'MB/sec write SST data time = 0.02's, 2671.521'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 129049160.784'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64716 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 1020.24 usrkeys = 64716 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0466 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 20.462870 GB avg = 0.267 KB (by entry num) total key len = 1.976122 GB avg = 0.026 KB total ukey len = 1.362652 GB avg = 0.018 KB total ukey num = 0.076683759 Billion total entry num = 0.076683759 Billion write speed all = 35.433454199 MB/sec (with seq num) write speed all = 34.464722492 MB/sec (without seq num) 2024/11/19-15:29:10.242478 139851179493120 [dcompact_etcd.cc:1163] job-00285/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.309 sec, input = 126.923 MiB 2024/11/19-15:29:10.260866 139851137550080 [compaction_job.cc:1948] [default] [JOB 278] Generated table #5468: 64716 keys, 66137412 bytes (need compaction), temperature: kUnknown 2024/11/19-15:29:10.260917 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001350260893, "cf_name": "default", "job": 278, "event": "table_file_creation", "file_number": 5468, "file_size": 66137412, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025616, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294320, "raw_average_key_size": 20, "raw_value_size": 65815072, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64716, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001350, "oldest_key_time": 0, "file_creation_time": 1732001350, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5468, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:29:10.275670 139851137550080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005469.sst) /storage/mytopling/..//tmp/Topling-82038-YIohyB 2024/11/19-15:29:10.328777 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005469.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:29:10.328871 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005469.sst): index pass time = 0.00's,7386.732'MB/sec index type = UintIndex_AllOne usrkeys = 63609 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:29:10.394118 139851137550080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005469.sst): first pass time = 0.12's, 556.974'MB/sec 2024/11/19-15:29:10.394169 139851137550080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005469.sst): old prealloc_size = 73819750, real_size = 64913816 2024/11/19-15:29:10.394192 139851137550080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005469.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63609 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0647 GB zip-val = 0.0649 GB avg-val =1017.00 avg-zval =1020.26 2024/11/19-15:29:10.447873 139851137550080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005469.sst): second pass time = 0.00's,244115543.396'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1208.838'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 9.863'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.414'MB/sec write SST data time = 0.02's, 4253.914'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 244115543.396'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63609 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.00 avg-zval = 1020.26 usrkeys = 63609 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.00 avg-zval = 1020.26 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0647 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0000 GB value = 0.0649 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index =3975.5625 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 20.527561 GB avg = 0.267 KB (by entry num) total key len = 1.977395 GB avg = 0.026 KB total ukey len = 1.363416 GB avg = 0.018 KB total ukey num = 0.076747368 Billion total entry num = 0.076747368 Billion write speed all = 35.525929961 MB/sec (with seq num) write speed all = 34.556713515 MB/sec (without seq num) 2024/11/19-15:29:10.451606 139851137550080 [compaction_job.cc:1948] [default] [JOB 278] Generated table #5469: 63609 keys, 64898890 bytes (need compaction), temperature: kUnknown 2024/11/19-15:29:10.451658 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001350451635, "cf_name": "default", "job": 278, "event": "table_file_creation", "file_number": 5469, "file_size": 64898890, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64897568, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1272180, "raw_average_key_size": 20, "raw_value_size": 64690619, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 63609, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001350, "oldest_key_time": 0, "file_creation_time": 1732001350, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5469, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:29:10.465933 139851137550080 [compaction_job.cc:703] job-00278: subcompact[0], size: 0.131036 G, files: /storage/mytopling/.rocksdb [005468,005469] 2024/11/19-15:29:10.466706 139851137550080 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 8329172 2024/11/19-15:29:10.466864 139851137550080 (Original Log Time 2024/11/19-15:29:10.050516) EVENT_LOG_v1 {"time_micros": 1732001350050498, "job": 278, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5283, 5285, 5287, 5290], "score": 5.92775, "input_data_size": 134098752, "oldest_snapshot_seqno": 156682599} 2024/11/19-15:29:10.466871 139851137550080 (Original Log Time 2024/11/19-15:29:10.466332) [compaction_job.cc:2022] [default] [JOB 278] Compacted 4@1 files to L2 => 131036302 bytes 2024/11/19-15:29:10.466875 139851137550080 (Original Log Time 2024/11/19-15:29:10.466674) [compaction_job.cc:1224] [default] compacted to: files[5 92 86 82 0 0 0] max score 4.18 (159 files need compaction), MB/sec: 322.8 rd, 315.4 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128325, records dropped: 0 output_compression: Snappy 2024/11/19-15:29:10.466896 139851137550080 (Original Log Time 2024/11/19-15:29:10.466699) EVENT_LOG_v1 {"time_micros": 1732001350466687, "cf": "default", "job": 278, "event": "compaction_finished", "compaction_time_micros": 415430, "compaction_time_cpu_micros": 392772, "output_level": 2, "num_output_files": 2, "total_output_size": 131036302, "num_input_records": 128325, "num_output_records": 128325, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 92, 86, 82, 0, 0, 0]} 2024/11/19-15:29:10.471129 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005346.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:10.471155 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001350471152, "job": 278, "event": "table_file_deletion", "file_number": 5346} 2024/11/19-15:29:10.475452 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005350.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:10.475475 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001350475472, "job": 278, "event": "table_file_deletion", "file_number": 5350} 2024/11/19-15:29:10.479734 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005355.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:10.479765 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001350479763, "job": 278, "event": "table_file_deletion", "file_number": 5355} 2024/11/19-15:29:10.483940 139851137550080 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005359.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:10.483961 139851137550080 EVENT_LOG_v1 {"time_micros": 1732001350483959, "job": 278, "event": "table_file_deletion", "file_number": 5359} 2024/11/19-15:29:10.484095 139851137550080 [compaction_job.cc:2437] [default] [JOB 286] Compacting 4@1 files to L2, score 4.18, subcompactions 1 : 1 2024/11/19-15:29:10.484104 139851137550080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 284 Base level 1, inputs: [5325(31MB) 5327(31MB) 5328(31MB) 5330(31MB)] 2024/11/19-15:29:10.688932 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00283/att-00 2024/11/19-15:29:10.689004 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00283/att-00/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:29:10.711209 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-02 2024/11/19-15:29:10.711268 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00279/att-02/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:29:10.711284 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00279/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.846 sec, real 11.487 (submit 277.980 ms wait 11.204 s), input 126.846 MiB, send shutdown 2024/11/19-15:29:10.716246 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00279/att-02 2024/11/19-15:29:10.716450 139851127064320 [dcompact_etcd.cc:1773] job-00279/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:10.718290 139851127064320 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00279) = success 2024/11/19-15:29:10.718303 139851127064320 [dcompact_etcd.cc:1773] job-00279/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:29:10.718322 139851127064320 (Original Log Time 2024/11/19-15:28:36.182446) EVENT_LOG_v1 {"time_micros": 1732001316182439, "job": 279, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5291, 5292, 5294, 5295], "score": 5.67797, "input_data_size": 134099456, "oldest_snapshot_seqno": -1} 2024/11/19-15:29:10.718329 139851127064320 [compaction_job.cc:2437] [default] [JOB 279] Compacting 4@1 files to L2, score 5.68, subcompactions 1 : 1 2024/11/19-15:29:10.718337 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 277 Base level 1, inputs: [5291(31MB) 5292(31MB) 5294(31MB) 5295(31MB)] 2024/11/19-15:29:10.718558 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005470.sst) /storage/mytopling/..//tmp/Topling-82038-1lNjvC 2024/11/19-15:29:10.771793 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005470.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:29:10.775128 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005470.sst): index pass time = 0.00's, 156.769'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64717 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:29:10.804176 139851137550080 [dcompact_etcd.cc:1163] job-00286/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.320 sec, input = 126.906 MiB 2024/11/19-15:29:10.839391 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005470.sst): first pass time = 0.12's, 555.453'MB/sec 2024/11/19-15:29:10.839437 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005470.sst): old prealloc_size = 73819750, real_size = 66152608 2024/11/19-15:29:10.839493 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005470.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64717 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.97 avg-zval =1020.23 2024/11/19-15:29:10.839535 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005470.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.068 sec, Key+Value = 0.067 GB 2024/11/19-15:29:10.900975 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005470.sst): second pass time = 0.00's,215788573.770'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1075.390'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2103.166'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.367'MB/sec write SST data time = 0.02's, 3805.227'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 215788573.770'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64717 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.97 avg-zval = 1020.23 usrkeys = 64717 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.97 avg-zval = 1020.23 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0467 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 20.593376 GB avg = 0.268 KB (by entry num) total key len = 1.978689 GB avg = 0.026 KB total ukey len = 1.364192 GB avg = 0.018 KB total ukey num = 0.076812085 Billion total entry num = 0.076812085 Billion write speed all = 35.606401670 MB/sec (with seq num) write speed all = 34.637061236 MB/sec (without seq num) 2024/11/19-15:29:10.918705 139851127064320 [compaction_job.cc:1948] [default] [JOB 279] Generated table #5470: 64717 keys, 66137860 bytes (need compaction), temperature: kUnknown 2024/11/19-15:29:10.918763 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001350918734, "cf_name": "default", "job": 279, "event": "table_file_creation", "file_number": 5470, "file_size": 66137860, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66026064, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294340, "raw_average_key_size": 20, "raw_value_size": 65815515, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64717, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001350, "oldest_key_time": 0, "file_creation_time": 1732001350, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5470, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:29:10.934027 139851127064320 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005471.sst) /storage/mytopling/..//tmp/Topling-82038-dzAQvc 2024/11/19-15:29:10.985815 139834024789760 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005471.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:29:10.990998 139834024789760 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005471.sst): index pass time = 0.01's, 98.765'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63547 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:29:11.053623 139851127064320 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005471.sst): first pass time = 0.12's, 551.089'MB/sec 2024/11/19-15:29:11.053677 139851127064320 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005471.sst): old prealloc_size = 73819750, real_size = 64958352 2024/11/19-15:29:11.053734 139851127064320 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005471.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63547 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0646 GB zip-val = 0.0648 GB avg-val =1017.00 avg-zval =1020.25 2024/11/19-15:29:11.053789 139851127064320 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005471.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.068 sec, Key+Value = 0.066 GB 2024/11/19-15:29:11.112510 139851127064320 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005471.sst): second pass time = 0.00's,224400152.778'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1104.192'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1733.662'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.379'MB/sec write SST data time = 0.02's, 3851.655'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 224400152.778'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63547 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.25 usrkeys = 63547 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7918 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0001 GB value = 0.0648 GB dict = 0.00 MB all = 0.0649 GB } UnZip/Zip{ index = 7.0420 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 20.658004 GB avg = 0.269 KB (by entry num) total key len = 1.979960 GB avg = 0.026 KB total ukey len = 1.364955 GB avg = 0.018 KB total ukey num = 0.076875632 Billion total entry num = 0.076875632 Billion write speed all = 35.698439916 MB/sec (with seq num) write speed all = 34.728621185 MB/sec (without seq num) 2024/11/19-15:29:11.116511 139851127064320 [compaction_job.cc:1948] [default] [JOB 279] Generated table #5471: 63547 keys, 64943812 bytes (need compaction), temperature: kUnknown 2024/11/19-15:29:11.116564 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001351116540, "cf_name": "default", "job": 279, "event": "table_file_creation", "file_number": 5471, "file_size": 64943812, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64833984, "index_size": 108288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1270940, "raw_average_key_size": 20, "raw_value_size": 64627244, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63547, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001350, "oldest_key_time": 0, "file_creation_time": 1732001350, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5471, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:29:11.131122 139851127064320 [compaction_job.cc:703] job-00279: subcompact[0], size: 0.131082 G, files: /storage/mytopling/.rocksdb [005470,005471] 2024/11/19-15:29:11.131915 139851127064320 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 8586775 2024/11/19-15:29:11.132072 139851127064320 (Original Log Time 2024/11/19-15:29:10.718364) EVENT_LOG_v1 {"time_micros": 1732001350718348, "job": 279, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5291, 5292, 5294, 5295], "score": 5.67797, "input_data_size": 134099456, "oldest_snapshot_seqno": -1} 2024/11/19-15:29:11.132079 139851127064320 (Original Log Time 2024/11/19-15:29:11.131536) [compaction_job.cc:2022] [default] [JOB 279] Compacted 4@1 files to L2 => 131081672 bytes 2024/11/19-15:29:11.132082 139851127064320 (Original Log Time 2024/11/19-15:29:11.131884) [compaction_job.cc:1224] [default] compacted to: files[5 88 88 82 0 0 0] max score 3.93 (157 files need compaction), MB/sec: 324.9 rd, 317.6 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128264, records dropped: 0 output_compression: Snappy 2024/11/19-15:29:11.132103 139851127064320 (Original Log Time 2024/11/19-15:29:11.131908) EVENT_LOG_v1 {"time_micros": 1732001351131896, "cf": "default", "job": 279, "event": "compaction_finished", "compaction_time_micros": 412770, "compaction_time_cpu_micros": 397078, "output_level": 2, "num_output_files": 2, "total_output_size": 131081672, "num_input_records": 128264, "num_output_records": 128264, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 88, 88, 82, 0, 0, 0]} 2024/11/19-15:29:11.136326 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005364.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:11.136350 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001351136347, "job": 279, "event": "table_file_deletion", "file_number": 5364} 2024/11/19-15:29:11.140630 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005367.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:11.140652 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001351140649, "job": 279, "event": "table_file_deletion", "file_number": 5367} 2024/11/19-15:29:11.144831 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005371.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:11.144852 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001351144850, "job": 279, "event": "table_file_deletion", "file_number": 5371} 2024/11/19-15:29:11.149615 139851127064320 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005374.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:11.149635 139851127064320 EVENT_LOG_v1 {"time_micros": 1732001351149633, "job": 279, "event": "table_file_deletion", "file_number": 5374} 2024/11/19-15:29:11.149774 139851127064320 [compaction_job.cc:2437] [default] [JOB 287] Compacting 4@1 files to L2, score 3.93, subcompactions 1 : 1 2024/11/19-15:29:11.149783 139851127064320 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 285 Base level 1, inputs: [5331(31MB) 5332(31MB) 5334(31MB) 5335(31MB)] 2024/11/19-15:29:11.413237 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00282/att-00 2024/11/19-15:29:11.413333 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00282/att-00/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:29:11.439657 139851127064320 [dcompact_etcd.cc:1163] job-00287/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.289 sec, input = 126.822 MiB 2024/11/19-15:29:11.573454 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-02 2024/11/19-15:29:11.573548 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00280/att-02/compact.done) = No such file or directory: 3.174 sec on http probe failed: 1, retry 2024/11/19-15:29:12.697197 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00283/att-00 2024/11/19-15:29:12.697271 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00283/att-00/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:29:12.838101 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00284/att-00 2024/11/19-15:29:12.838175 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00284/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:13.423061 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00282/att-00 2024/11/19-15:29:13.423137 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00282/att-00/compact.done) = No such file or directory: 9.194 sec on http probe failed: 4, retry 2024/11/19-15:29:13.424268 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00285/att-00 2024/11/19-15:29:13.424324 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00285/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:13.579581 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-02 2024/11/19-15:29:13.579651 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00280/att-02/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:29:13.987067 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00286/att-00 2024/11/19-15:29:13.987149 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00286/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:14.615841 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00287/att-00 2024/11/19-15:29:14.615935 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00287/att-00/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:29:14.705467 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00283/att-00 2024/11/19-15:29:14.705544 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00283/att-00/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:29:14.843746 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00284/att-00 2024/11/19-15:29:14.843814 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00284/att-00/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:29:15.428683 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00282/att-00 2024/11/19-15:29:15.428757 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00282/att-00/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:29:15.428775 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00282/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.808 sec, real 11.510 (submit 301.044 ms wait 11.204 s), input 126.808 MiB, send shutdown 2024/11/19-15:29:15.432620 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00285/att-00 2024/11/19-15:29:15.432689 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00285/att-00/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:29:15.433726 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00282/att-00 2024/11/19-15:29:15.433944 139851158521600 [dcompact_etcd.cc:1773] job-00282/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:15.587561 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-02 2024/11/19-15:29:15.587638 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00280/att-02/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:29:15.714033 139851158521600 [dcompact_etcd.cc:1163] job-00282/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.280 sec, input = 126.808 MiB 2024/11/19-15:29:15.996353 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00286/att-00 2024/11/19-15:29:15.996428 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00286/att-00/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:29:16.625181 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00287/att-00 2024/11/19-15:29:16.625260 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00287/att-00/compact.done) = No such file or directory: 5.176 sec on http probe failed: 2, retry 2024/11/19-15:29:16.711187 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00283/att-00 2024/11/19-15:29:16.711252 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00283/att-00/compact.done) = No such file or directory: 11.205 sec on http probe failed: 5, giveup 2024/11/19-15:29:16.711269 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00283/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.914 sec, real 11.491 (submit 280.359 ms wait 11.205 s), input 126.914 MiB, send shutdown 2024/11/19-15:29:16.720683 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00283/att-00 2024/11/19-15:29:16.720896 139850978166528 [dcompact_etcd.cc:1773] job-00283/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:16.849100 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00284/att-00 2024/11/19-15:29:16.849187 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00284/att-00/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:29:16.993890 139850978166528 [dcompact_etcd.cc:1163] job-00283/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.273 sec, input = 126.914 MiB 2024/11/19-15:29:17.441805 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00285/att-00 2024/11/19-15:29:17.441880 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00285/att-00/compact.done) = No such file or directory: 7.190 sec on http probe failed: 3, retry 2024/11/19-15:29:17.596080 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-02 2024/11/19-15:29:17.596148 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00280/att-02/compact.done) = No such file or directory: 9.194 sec on http probe failed: 4, retry 2024/11/19-15:29:18.005010 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00286/att-00 2024/11/19-15:29:18.005098 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00286/att-00/compact.done) = No such file or directory: 7.192 sec on http probe failed: 3, retry 2024/11/19-15:29:18.633622 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00287/att-00 2024/11/19-15:29:18.633700 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00287/att-00/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:29:18.854646 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00284/att-00 2024/11/19-15:29:18.854726 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00284/att-00/compact.done) = No such file or directory: 9.192 sec on http probe failed: 4, retry 2024/11/19-15:29:18.893058 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00282/att-01 2024/11/19-15:29:18.893124 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00282/att-01/compact.done) = No such file or directory: 3.170 sec on http probe failed: 1, retry 2024/11/19-15:29:19.447760 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00285/att-00 2024/11/19-15:29:19.447843 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00285/att-00/compact.done) = No such file or directory: 9.199 sec on http probe failed: 4, retry 2024/11/19-15:29:19.551971 139852349699840 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 8329171 2024/11/19-15:29:19.605141 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-02 2024/11/19-15:29:19.605210 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00280/att-02/compact.done) = No such file or directory: 11.203 sec on http probe failed: 5, giveup 2024/11/19-15:29:19.605228 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00280/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.971 sec, real 11.496 (submit 284.481 ms wait 11.203 s), input 126.971 MiB, send shutdown 2024/11/19-15:29:19.613480 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00280/att-02 2024/11/19-15:29:19.613701 139851169007360 [dcompact_etcd.cc:1773] job-00280/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:19.615835 139851169007360 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00280) = success 2024/11/19-15:29:19.615843 139851169007360 [dcompact_etcd.cc:1773] job-00280/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:29:19.615860 139851169007360 (Original Log Time 2024/11/19-15:28:45.078190) EVENT_LOG_v1 {"time_micros": 1732001325078182, "job": 280, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5296, 5298, 5299, 5301], "score": 5.42819, "input_data_size": 134099392, "oldest_snapshot_seqno": -1} 2024/11/19-15:29:19.615868 139851169007360 [compaction_job.cc:2437] [default] [JOB 280] Compacting 4@1 files to L2, score 5.43, subcompactions 1 : 1 2024/11/19-15:29:19.615877 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 278 Base level 1, inputs: [5296(31MB) 5298(31MB) 5299(31MB) 5301(31MB)] 2024/11/19-15:29:19.616098 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005472.sst) /storage/mytopling/..//tmp/Topling-82038-Pzo42V 2024/11/19-15:29:19.669082 139834056247040 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005472.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:29:19.669192 139834056247040 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005472.sst): index pass time = 0.00's,6243.157'MB/sec index type = UintIndex_AllOne usrkeys = 64715 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:29:19.735675 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005472.sst): first pass time = 0.12's, 561.279'MB/sec 2024/11/19-15:29:19.735721 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005472.sst): old prealloc_size = 73819750, real_size = 66041912 2024/11/19-15:29:19.735743 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005472.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 64715 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1017.00 avg-zval =1020.25 2024/11/19-15:29:19.789992 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005472.sst): second pass time = 0.00's,300524525.114'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1217.096'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.199'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.417'MB/sec write SST data time = 0.02's, 4272.025'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 300524525.114'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64715 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1017.00 avg-zval = 1020.25 usrkeys = 64715 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1017.00 avg-zval = 1020.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0000 GB value = 0.0660 GB dict = 0.00 MB all = 0.0660 GB } UnZip/Zip{ index =4044.6875 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0002 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 20.723818 GB avg = 0.269 KB (by entry num) total key len = 1.981254 GB avg = 0.026 KB total ukey len = 1.365731 GB avg = 0.018 KB total ukey num = 0.076940347 Billion total entry num = 0.076940347 Billion write speed all = 35.320942865 MB/sec (with seq num) write speed all = 34.363410374 MB/sec (without seq num) 2024/11/19-15:29:19.793692 139851169007360 [compaction_job.cc:1948] [default] [JOB 280] Generated table #5472: 64715 keys, 66026698 bytes (need compaction), temperature: kUnknown 2024/11/19-15:29:19.793771 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001359793722, "cf_name": "default", "job": 280, "event": "table_file_creation", "file_number": 5472, "file_size": 66026698, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025408, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294300, "raw_average_key_size": 20, "raw_value_size": 65814871, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64715, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001359, "oldest_key_time": 0, "file_creation_time": 1732001359, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5472, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:29:19.808053 139851169007360 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005473.sst) /storage/mytopling/..//tmp/Topling-82038-MtKxC9 2024/11/19-15:29:19.859429 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005473.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:29:19.862724 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005473.sst): index pass time = 0.00's, 156.417'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 63674 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:29:19.924835 139851169007360 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005473.sst): first pass time = 0.12's, 565.487'MB/sec 2024/11/19-15:29:19.924883 139851169007360 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005473.sst): old prealloc_size = 73819750, real_size = 65087888 2024/11/19-15:29:19.924947 139851169007360 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005473.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 63674 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0648 GB zip-val = 0.0650 GB avg-val =1017.00 avg-zval =1020.25 2024/11/19-15:29:19.925000 139851169007360 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005473.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.066 sec, Key+Value = 0.066 GB 2024/11/19-15:29:20.011067 139851169007360 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005473.sst): second pass time = 0.00's,291694441.441'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.09's, 755.199'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1769.109'MB/sec (index lex order gen) rebuild zvType time = 0.07's, 0.231'MB/sec write SST data time = 0.02's, 3792.490'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 291694441.441'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63674 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.25 usrkeys = 63674 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.00 avg-zval = 1020.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7934 UnZipSize{ index = 0.0008 GB value = 0.0648 GB dict = 0.00 MB all = 0.0660 GB } __ZipSize{ index = 0.0001 GB value = 0.0650 GB dict = 0.00 MB all = 0.0651 GB } UnZip/Zip{ index = 7.0436 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1420 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 20.788575 GB avg = 0.270 KB (by entry num) total key len = 1.982528 GB avg = 0.026 KB total ukey len = 1.366496 GB avg = 0.018 KB total ukey num = 0.077004021 Billion total entry num = 0.077004021 Billion write speed all = 35.411482598 MB/sec (with seq num) write speed all = 34.453487149 MB/sec (without seq num) 2024/11/19-15:29:20.013394 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00286/att-00 2024/11/19-15:29:20.013497 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00286/att-00/compact.done) = No such file or directory: 9.201 sec on http probe failed: 4, retry 2024/11/19-15:29:20.060012 139851169007360 [compaction_job.cc:1948] [default] [JOB 280] Generated table #5473: 63674 keys, 65073348 bytes (need compaction), temperature: kUnknown 2024/11/19-15:29:20.060064 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001360060040, "cf_name": "default", "job": 280, "event": "table_file_creation", "file_number": 5473, "file_size": 65073348, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64963328, "index_size": 108480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1273480, "raw_average_key_size": 20, "raw_value_size": 64756166, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63674, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001359, "oldest_key_time": 0, "file_creation_time": 1732001359, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5473, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:29:20.074333 139851169007360 [compaction_job.cc:703] job-00280: subcompact[0], size: 0.131100 G, files: /storage/mytopling/.rocksdb [005472,005473] 2024/11/19-15:29:20.075070 139851169007360 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 8586774 2024/11/19-15:29:20.075203 139851169007360 (Original Log Time 2024/11/19-15:29:19.615902) EVENT_LOG_v1 {"time_micros": 1732001359615888, "job": 280, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5296, 5298, 5299, 5301], "score": 5.42819, "input_data_size": 134099392, "oldest_snapshot_seqno": -1} 2024/11/19-15:29:20.075209 139851169007360 (Original Log Time 2024/11/19-15:29:20.074697) [compaction_job.cc:2022] [default] [JOB 280] Compacted 4@1 files to L2 => 131100046 bytes 2024/11/19-15:29:20.075213 139851169007360 (Original Log Time 2024/11/19-15:29:20.075040) [compaction_job.cc:1224] [default] compacted to: files[5 84 90 82 0 0 0] max score 3.68 (155 files need compaction), MB/sec: 292.5 rd, 286.0 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(125.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128389, records dropped: 0 output_compression: Snappy 2024/11/19-15:29:20.075229 139851169007360 (Original Log Time 2024/11/19-15:29:20.075064) EVENT_LOG_v1 {"time_micros": 1732001360075053, "cf": "default", "job": 280, "event": "compaction_finished", "compaction_time_micros": 458444, "compaction_time_cpu_micros": 384186, "output_level": 2, "num_output_files": 2, "total_output_size": 131100046, "num_input_records": 128389, "num_output_records": 128389, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 84, 90, 82, 0, 0, 0]} 2024/11/19-15:29:20.088885 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005252.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:20.088909 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001360088906, "job": 280, "event": "table_file_deletion", "file_number": 5252} 2024/11/19-15:29:20.093501 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005255.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:20.093521 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001360093519, "job": 280, "event": "table_file_deletion", "file_number": 5255} 2024/11/19-15:29:20.097604 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005380.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:20.097623 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001360097621, "job": 280, "event": "table_file_deletion", "file_number": 5380} 2024/11/19-15:29:20.100774 139851169007360 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005381.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:20.100793 139851169007360 EVENT_LOG_v1 {"time_micros": 1732001360100791, "job": 280, "event": "table_file_deletion", "file_number": 5381} 2024/11/19-15:29:20.100965 139851169007360 [compaction_job.cc:2437] [default] [JOB 288] Compacting 4@1 files to L2, score 3.68, subcompactions 1 : 1 2024/11/19-15:29:20.100974 139851169007360 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 286 Base level 1, inputs: [5337(31MB) 5338(31MB) 5340(31MB) 5341(31MB)] 2024/11/19-15:29:20.173115 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00283/att-01 2024/11/19-15:29:20.173197 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00283/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:20.417680 139851169007360 [dcompact_etcd.cc:1163] job-00288/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.316 sec, input = 126.911 MiB 2024/11/19-15:29:20.638476 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00287/att-00 2024/11/19-15:29:20.638576 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00287/att-00/compact.done) = No such file or directory: 9.194 sec on http probe failed: 4, retry 2024/11/19-15:29:20.860732 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00284/att-00 2024/11/19-15:29:20.860809 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00284/att-00/compact.done) = No such file or directory: 11.198 sec on http probe failed: 5, giveup 2024/11/19-15:29:20.860829 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00284/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.909 sec, real 11.505 (submit 301.330 ms wait 11.198 s), input 126.909 MiB, send shutdown 2024/11/19-15:29:20.868623 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00284/att-00 2024/11/19-15:29:20.868858 139851148035840 [dcompact_etcd.cc:1773] job-00284/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:20.901187 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00282/att-01 2024/11/19-15:29:20.901243 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00282/att-01/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:29:21.165758 139851148035840 [dcompact_etcd.cc:1163] job-00284/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.297 sec, input = 126.909 MiB 2024/11/19-15:29:21.456562 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00285/att-00 2024/11/19-15:29:21.456630 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00285/att-00/compact.done) = No such file or directory: 11.205 sec on http probe failed: 5, giveup 2024/11/19-15:29:21.456648 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00285/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.923 sec, real 11.523 (submit 309.171 ms wait 11.205 s), input 126.923 MiB, send shutdown 2024/11/19-15:29:21.464424 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00285/att-00 2024/11/19-15:29:21.464641 139851179493120 [dcompact_etcd.cc:1773] job-00285/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:21.752682 139851179493120 [dcompact_etcd.cc:1163] job-00285/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.288 sec, input = 126.923 MiB 2024/11/19-15:29:22.022412 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00286/att-00 2024/11/19-15:29:22.022487 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00286/att-00/compact.done) = No such file or directory: 11.209 sec on http probe failed: 5, giveup 2024/11/19-15:29:22.022506 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00286/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.906 sec, real 11.538 (submit 319.657 ms wait 11.209 s), input 126.906 MiB, send shutdown 2024/11/19-15:29:22.031330 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00286/att-00 2024/11/19-15:29:22.031552 139851137550080 [dcompact_etcd.cc:1773] job-00286/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:22.181980 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00283/att-01 2024/11/19-15:29:22.182038 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00283/att-01/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:29:22.321935 139851137550080 [dcompact_etcd.cc:1163] job-00286/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.290 sec, input = 126.906 MiB 2024/11/19-15:29:22.647106 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00287/att-00 2024/11/19-15:29:22.647181 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00287/att-00/compact.done) = No such file or directory: 11.199 sec on http probe failed: 5, giveup 2024/11/19-15:29:22.647199 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00287/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.822 sec, real 11.497 (submit 289.418 ms wait 11.199 s), input 126.822 MiB, send shutdown 2024/11/19-15:29:22.652855 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00287/att-00 2024/11/19-15:29:22.653087 139851127064320 [dcompact_etcd.cc:1773] job-00287/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:22.906778 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00282/att-01 2024/11/19-15:29:22.906841 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00282/att-01/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:29:22.937421 139851127064320 [dcompact_etcd.cc:1163] job-00287/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.284 sec, input = 126.822 MiB 2024/11/19-15:29:23.596313 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00288/att-00 2024/11/19-15:29:23.596402 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00288/att-00/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:24.191258 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00283/att-01 2024/11/19-15:29:24.191333 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00283/att-01/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:29:24.343799 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00284/att-01 2024/11/19-15:29:24.343880 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00284/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:24.913573 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00282/att-01 2024/11/19-15:29:24.913663 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00282/att-01/compact.done) = No such file or directory: 9.193 sec on http probe failed: 4, retry 2024/11/19-15:29:24.936085 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00285/att-01 2024/11/19-15:29:24.936158 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00285/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:25.504266 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00286/att-01 2024/11/19-15:29:25.504353 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00286/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:25.604753 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00288/att-00 2024/11/19-15:29:25.604822 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00288/att-00/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:29:26.113062 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00287/att-01 2024/11/19-15:29:26.113159 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00287/att-01/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:29:26.200347 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00283/att-01 2024/11/19-15:29:26.200433 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00283/att-01/compact.done) = No such file or directory: 9.198 sec on http probe failed: 4, retry 2024/11/19-15:29:26.354211 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00284/att-01 2024/11/19-15:29:26.354285 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00284/att-01/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:29:26.923118 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00282/att-01 2024/11/19-15:29:26.923205 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00282/att-01/compact.done) = No such file or directory: 11.200 sec on http probe failed: 5, giveup 2024/11/19-15:29:26.923223 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00282/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.808 sec, real 11.489 (submit 280.000 ms wait 11.200 s), input 126.808 MiB, send shutdown 2024/11/19-15:29:26.932663 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00282/att-01 2024/11/19-15:29:26.932889 139851158521600 [dcompact_etcd.cc:1773] job-00282/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:26.944510 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00285/att-01 2024/11/19-15:29:26.944563 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00285/att-01/compact.done) = No such file or directory: 5.184 sec on http probe failed: 2, retry 2024/11/19-15:29:27.211711 139851158521600 [dcompact_etcd.cc:1163] job-00282/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.279 sec, input = 126.808 MiB 2024/11/19-15:29:27.509319 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00286/att-01 2024/11/19-15:29:27.509395 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00286/att-01/compact.done) = No such file or directory: 5.183 sec on http probe failed: 2, retry 2024/11/19-15:29:27.611216 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00288/att-00 2024/11/19-15:29:27.611281 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00288/att-00/compact.done) = No such file or directory: 7.187 sec on http probe failed: 3, retry 2024/11/19-15:29:28.118882 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00287/att-01 2024/11/19-15:29:28.118982 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00287/att-01/compact.done) = No such file or directory: 5.176 sec on http probe failed: 2, retry 2024/11/19-15:29:28.209483 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00283/att-01 2024/11/19-15:29:28.209549 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00283/att-01/compact.done) = No such file or directory: 11.207 sec on http probe failed: 5, giveup 2024/11/19-15:29:28.209566 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00283/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.914 sec, real 11.489 (submit 272.900 ms wait 11.207 s), input 126.914 MiB, send shutdown 2024/11/19-15:29:28.217765 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00283/att-01 2024/11/19-15:29:28.217993 139850978166528 [dcompact_etcd.cc:1773] job-00283/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:28.360552 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00284/att-01 2024/11/19-15:29:28.360615 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00284/att-01/compact.done) = No such file or directory: 7.189 sec on http probe failed: 3, retry 2024/11/19-15:29:28.494647 139850978166528 [dcompact_etcd.cc:1163] job-00283/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.277 sec, input = 126.914 MiB 2024/11/19-15:29:28.949350 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00285/att-01 2024/11/19-15:29:28.949431 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00285/att-01/compact.done) = No such file or directory: 7.192 sec on http probe failed: 3, retry 2024/11/19-15:29:29.518592 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00286/att-01 2024/11/19-15:29:29.518690 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00286/att-01/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:29:29.619477 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00288/att-00 2024/11/19-15:29:29.619545 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00288/att-00/compact.done) = No such file or directory: 9.194 sec on http probe failed: 4, retry 2024/11/19-15:29:30.127793 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00287/att-01 2024/11/19-15:29:30.127879 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00287/att-01/compact.done) = No such file or directory: 7.182 sec on http probe failed: 3, retry 2024/11/19-15:29:30.369583 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00284/att-01 2024/11/19-15:29:30.369662 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00284/att-01/compact.done) = No such file or directory: 9.195 sec on http probe failed: 4, retry 2024/11/19-15:29:30.391135 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00282/att-02 2024/11/19-15:29:30.391205 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00282/att-02/compact.done) = No such file or directory: 3.170 sec on http probe failed: 1, retry 2024/11/19-15:29:30.957979 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00285/att-01 2024/11/19-15:29:30.958055 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00285/att-01/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:29:31.527596 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00286/att-01 2024/11/19-15:29:31.527684 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00286/att-01/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:29:31.628202 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00288/att-00 2024/11/19-15:29:31.628270 139851169007360 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00288/att-00/compact.done) = No such file or directory: 11.202 sec on http probe failed: 5, giveup 2024/11/19-15:29:31.628288 139851169007360 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00288/att-00: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.911 sec, real 11.527 (submit 316.313 ms wait 11.202 s), input 126.911 MiB, send shutdown 2024/11/19-15:29:31.636227 139851169007360 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00288/att-00 2024/11/19-15:29:31.636446 139851169007360 [dcompact_etcd.cc:1773] job-00288/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:31.673436 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00283/att-02 2024/11/19-15:29:31.673499 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00283/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:31.902640 139851169007360 [dcompact_etcd.cc:1163] job-00288/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.266 sec, input = 126.911 MiB 2024/11/19-15:29:32.135957 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00287/att-01 2024/11/19-15:29:32.136055 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00287/att-01/compact.done) = No such file or directory: 9.191 sec on http probe failed: 4, retry 2024/11/19-15:29:32.374487 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00284/att-01 2024/11/19-15:29:32.374560 139851148035840 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00284/att-01/compact.done) = No such file or directory: 11.204 sec on http probe failed: 5, giveup 2024/11/19-15:29:32.374578 139851148035840 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00284/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.909 sec, real 11.506 (submit 296.814 ms wait 11.204 s), input 126.909 MiB, send shutdown 2024/11/19-15:29:32.383436 139851148035840 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00284/att-01 2024/11/19-15:29:32.383653 139851148035840 [dcompact_etcd.cc:1773] job-00284/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:32.399174 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00282/att-02 2024/11/19-15:29:32.399228 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00282/att-02/compact.done) = No such file or directory: 5.180 sec on http probe failed: 2, retry 2024/11/19-15:29:32.644086 139851148035840 [dcompact_etcd.cc:1163] job-00284/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.260 sec, input = 126.909 MiB 2024/11/19-15:29:32.963677 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00285/att-01 2024/11/19-15:29:32.963751 139851179493120 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00285/att-01/compact.done) = No such file or directory: 11.205 sec on http probe failed: 5, giveup 2024/11/19-15:29:32.963769 139851179493120 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00285/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.923 sec, real 11.499 (submit 287.965 ms wait 11.205 s), input 126.923 MiB, send shutdown 2024/11/19-15:29:32.973775 139851179493120 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00285/att-01 2024/11/19-15:29:32.974087 139851179493120 [dcompact_etcd.cc:1773] job-00285/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:33.282175 139851179493120 [dcompact_etcd.cc:1163] job-00285/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.308 sec, input = 126.923 MiB 2024/11/19-15:29:33.532583 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00286/att-01 2024/11/19-15:29:33.532651 139851137550080 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00286/att-01/compact.done) = No such file or directory: 11.206 sec on http probe failed: 5, giveup 2024/11/19-15:29:33.532669 139851137550080 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00286/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.906 sec, real 11.501 (submit 290.303 ms wait 11.206 s), input 126.906 MiB, send shutdown 2024/11/19-15:29:33.537958 139851137550080 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00286/att-01 2024/11/19-15:29:33.538166 139851137550080 [dcompact_etcd.cc:1773] job-00286/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:33.682402 139850978166528 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00283/att-02 2024/11/19-15:29:33.682457 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00283/att-02/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:29:33.793740 139851137550080 [dcompact_etcd.cc:1163] job-00286/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.256 sec, input = 126.906 MiB 2024/11/19-15:29:34.140910 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00287/att-01 2024/11/19-15:29:34.141009 139851127064320 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00287/att-01/compact.done) = No such file or directory: 11.199 sec on http probe failed: 5, giveup 2024/11/19-15:29:34.141028 139851127064320 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00287/att-01: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.822 sec, real 11.488 (submit 284.256 ms wait 11.199 s), input 126.822 MiB, send shutdown 2024/11/19-15:29:34.145664 139851127064320 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00287/att-01 2024/11/19-15:29:34.145882 139851127064320 [dcompact_etcd.cc:1773] job-00287/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:34.405816 139851127064320 [dcompact_etcd.cc:1163] job-00287/att-02: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.260 sec, input = 126.822 MiB 2024/11/19-15:29:34.408348 139851158521600 [dcompact_etcd.cc:1500] curl http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : 200 OK status = NotFound : job-00282/att-02 2024/11/19-15:29:34.408409 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00282/att-02/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:29:35.080629 139851169007360 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00288/att-01 : response_code = 502 bad gateway 2024/11/19-15:29:35.080752 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00288/att-01/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:35.690293 139850978166528 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00283/att-02 : response_code = 502 bad gateway 2024/11/19-15:29:35.690423 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00283/att-02/compact.done) = No such file or directory: 7.188 sec on http probe failed: 3, retry 2024/11/19-15:29:35.822937 139851148035840 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00284/att-02 : response_code = 502 bad gateway 2024/11/19-15:29:35.823036 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00284/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:36.417937 139851158521600 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00282/att-02 : response_code = 502 bad gateway 2024/11/19-15:29:36.418049 139851158521600 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00282/att-02/compact.done) = No such file or directory: 9.197 sec on http probe failed: 4, retry 2024/11/19-15:29:36.463073 139851179493120 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00285/att-02 : response_code = 502 bad gateway 2024/11/19-15:29:36.463200 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00285/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:36.975840 139851137550080 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00286/att-02 : response_code = 502 bad gateway 2024/11/19-15:29:36.975993 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00286/att-02/compact.done) = No such file or directory: 3.173 sec on http probe failed: 1, retry 2024/11/19-15:29:37.088749 139851169007360 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00288/att-01 : response_code = 502 bad gateway 2024/11/19-15:29:37.088843 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00288/att-01/compact.done) = No such file or directory: 5.178 sec on http probe failed: 2, retry 2024/11/19-15:29:37.584182 139851127064320 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00287/att-02 : response_code = 502 bad gateway 2024/11/19-15:29:37.584304 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00287/att-02/compact.done) = No such file or directory: 3.171 sec on http probe failed: 1, retry 2024/11/19-15:29:37.698967 139850978166528 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00283/att-02 : response_code = 502 bad gateway 2024/11/19-15:29:37.699076 139850978166528 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00283/att-02/compact.done) = No such file or directory: 9.196 sec on http probe failed: 4, retry 2024/11/19-15:29:37.831550 139851148035840 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00284/att-02 : response_code = 502 bad gateway 2024/11/19-15:29:37.831643 139851148035840 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00284/att-02/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:29:38.425470 139851158521600 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00282/att-02 : response_code = 502 bad gateway 2024/11/19-15:29:38.425596 139851158521600 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00282/att-02/compact.done) = No such file or directory: 11.206 sec on http probe failed: 5, giveup 2024/11/19-15:29:38.425614 139851158521600 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00282/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.808 sec, real 11.493 (submit 278.730 ms wait 11.206 s), input 126.808 MiB, send shutdown 2024/11/19-15:29:38.435989 139851158521600 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00282/att-02 : response_code = 502 bad gateway 2024/11/19-15:29:38.436239 139851158521600 [dcompact_etcd.cc:1773] job-00282/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:38.438095 139851158521600 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00282) = success 2024/11/19-15:29:38.438102 139851158521600 [dcompact_etcd.cc:1773] job-00282/att-03: CleanFiles 0.002 sec : 0.000 2024/11/19-15:29:38.438122 139851158521600 (Original Log Time 2024/11/19-15:29:03.918111) EVENT_LOG_v1 {"time_micros": 1732001343918105, "job": 282, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5302, 5304, 5306, 5308], "score": 5.17841, "input_data_size": 134099328, "oldest_snapshot_seqno": 158445087} 2024/11/19-15:29:38.438129 139851158521600 [compaction_job.cc:2437] [default] [JOB 282] Compacting 4@1 files to L2, score 5.18, subcompactions 1 : 1 2024/11/19-15:29:38.438137 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 280 Base level 1, inputs: [5302(31MB) 5304(31MB) 5306(31MB) 5308(31MB)] 2024/11/19-15:29:38.438351 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005474.sst) /storage/mytopling/..//tmp/Topling-82038-QVjj91 2024/11/19-15:29:38.468787 139851179493120 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00285/att-02 : response_code = 502 bad gateway 2024/11/19-15:29:38.468885 139851179493120 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00285/att-02/compact.done) = No such file or directory: 5.181 sec on http probe failed: 2, retry 2024/11/19-15:29:38.492106 139834045761280 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005474.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/19-15:29:38.497382 139834045761280 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005474.sst): index pass time = 0.01's, 98.663'MB/sec index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL usrkeys = 64716 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 3 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 2024/11/19-15:29:38.559021 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005474.sst): first pass time = 0.12's, 556.203'MB/sec 2024/11/19-15:29:38.559072 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005474.sst): old prealloc_size = 73819750, real_size = 66152192 2024/11/19-15:29:38.559127 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005474.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = PlainBlobStore usrkeys = 64716 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0001 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.0658 GB zip-val = 0.0660 GB avg-val =1016.98 avg-zval =1020.24 2024/11/19-15:29:38.559167 139851158521600 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005474.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.067 sec, Key+Value = 0.067 GB 2024/11/19-15:29:38.618079 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005474.sst): second pass time = 0.00's,224624897.611'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.06's, 1121.162'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2140.501'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.383'MB/sec write SST data time = 0.02's, 3958.460'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.066'GB zip my value throughput = 224624897.611'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 64716 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 1020.24 usrkeys = 64716 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8064 UnZipSize{ index = 0.0008 GB value = 0.0658 GB dict = 0.00 MB all = 0.0671 GB } __ZipSize{ index = 0.0001 GB value = 0.0660 GB dict = 0.00 MB all = 0.0661 GB } UnZip/Zip{ index = 7.0466 value = 0.9968 dict = 0.00 all = 1.0147 } Zip/UnZip{ index = 0.1419 value = 1.0032 dict = 0.00 all = 0.9855 } ---------------------------- total value len = 20.854390 GB avg = 0.271 KB (by entry num) total key len = 1.983822 GB avg = 0.026 KB total ukey len = 1.367272 GB avg = 0.018 KB total ukey num = 0.077068737 Billion total entry num = 0.077068737 Billion write speed all = 34.517063309 MB/sec (with seq num) write speed all = 33.585226262 MB/sec (without seq num) 2024/11/19-15:29:38.621614 139851158521600 [compaction_job.cc:1948] [default] [JOB 282] Generated table #5474: 64716 keys, 66137412 bytes (need compaction), temperature: kUnknown 2024/11/19-15:29:38.621667 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001378621643, "cf_name": "default", "job": 282, "event": "table_file_creation", "file_number": 5474, "file_size": 66137412, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 66025648, "index_size": 110208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1294320, "raw_average_key_size": 20, "raw_value_size": 65815095, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 64716, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-PlainBlobStore", "creation_time": 1732001378, "oldest_key_time": 0, "file_creation_time": 1732001378, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5474, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:29:38.635754 139851158521600 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/mytopling/.rocksdb/005475.sst) /storage/mytopling/..//tmp/Topling-82038-R2Z9Kj 2024/11/19-15:29:38.687449 139834066732800 [top_zip_table_builder.cc:1325] /storage/mytopling/.rocksdb/005475.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, nltTrie workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.066 GB 2024/11/19-15:29:38.687555 139834066732800 [top_zip_table_builder.cc:1588] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005475.sst): index pass time = 0.00's,6453.101'MB/sec index type = UintIndex_AllOne usrkeys = 63509 min-keylen = 8 max-keylen = 8 prefix = 4 subcplen = 5 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 2024/11/19-15:29:38.751759 139851158521600 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005475.sst): first pass time = 0.12's, 567.799'MB/sec 2024/11/19-15:29:38.751807 139851158521600 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005475.sst): old prealloc_size = 73819750, real_size = 64810968 2024/11/19-15:29:38.751828 139851158521600 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/mytopling/.rocksdb/005475.sst): index type = UintIndex_AllOne , store type = PlainBlobStore usrkeys = 63509 prefix = 4 raw-key = 0.0005 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.00 raw-val = 0.0646 GB zip-val = 0.0648 GB avg-val =1016.99 avg-zval =1020.24 2024/11/19-15:29:38.804445 139851158521600 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/mytopling/.rocksdb/005475.sst): second pass time = 0.00's,294922817.352'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 1231.374'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 10.328'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.423'MB/sec write SST data time = 0.02's, 4303.470'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.065'GB zip my value throughput = 294922817.352'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 63509 avg-key = 20.00 avg-zkey = 0.00 avg-val = 1016.99 avg-zval = 1020.24 usrkeys = 63509 avg-key = 12.00 avg-zkey = 0.00 avg-val = 1016.99 avg-zval = 1020.24 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0008 GB value = 0.0646 GB dict = 0.00 MB all = 0.0659 GB } __ZipSize{ index = 0.0000 GB value = 0.0648 GB dict = 0.00 MB all = 0.0648 GB } UnZip/Zip{ index =3969.3125 value = 0.9968 dict = 0.00 all = 1.0164 } Zip/UnZip{ index = 0.0003 value = 1.0032 dict = 0.00 all = 0.9839 } ---------------------------- total value len = 20.918978 GB avg = 0.271 KB (by entry num) total key len = 1.985092 GB avg = 0.026 KB total ukey len = 1.368034 GB avg = 0.018 KB total ukey num = 0.077132246 Billion total entry num = 0.077132246 Billion write speed all = 34.606852073 MB/sec (with seq num) write speed all = 33.674509752 MB/sec (without seq num) 2024/11/19-15:29:38.808099 139851158521600 [compaction_job.cc:1948] [default] [JOB 282] Generated table #5475: 63509 keys, 64796042 bytes (need compaction), temperature: kUnknown 2024/11/19-15:29:38.808153 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001378808129, "cf_name": "default", "job": 282, "event": "table_file_creation", "file_number": 5475, "file_size": 64796042, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 64794720, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1270180, "raw_average_key_size": 20, "raw_value_size": 64588097, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 63509, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-UintIndex_AllOne-PlainBlobStore", "creation_time": 1732001378, "oldest_key_time": 0, "file_creation_time": 1732001378, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MU9ZNZIP7NRMU2YPYQ8B", "orig_file_number": 5475, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/19-15:29:38.822002 139851158521600 [compaction_job.cc:703] job-00282: subcompact[0], size: 0.130933 G, files: /storage/mytopling/.rocksdb [005474,005475] 2024/11/19-15:29:38.822774 139851158521600 [WARN] [column_family.cc:1044] [default] Stalling writes because we have 5 level-0 files rate 8852344 2024/11/19-15:29:38.822908 139851158521600 (Original Log Time 2024/11/19-15:29:38.438165) EVENT_LOG_v1 {"time_micros": 1732001378438150, "job": 282, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5302, 5304, 5306, 5308], "score": 5.17841, "input_data_size": 134099328, "oldest_snapshot_seqno": 158445087} 2024/11/19-15:29:38.822914 139851158521600 (Original Log Time 2024/11/19-15:29:38.822402) [compaction_job.cc:2022] [default] [JOB 282] Compacted 4@1 files to L2 => 130933454 bytes 2024/11/19-15:29:38.822918 139851158521600 (Original Log Time 2024/11/19-15:29:38.822745) [compaction_job.cc:1224] [default] compacted to: files[5 80 92 82 0 0 0] max score 3.50 (153 files need compaction), MB/sec: 349.4 rd, 341.1 wr, level 2, files in(4, 0) out(2 +0 blob) MB in(127.9, 0.0 +0.0 blob) out(124.9 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 128225, records dropped: 0 output_compression: Snappy 2024/11/19-15:29:38.822947 139851158521600 (Original Log Time 2024/11/19-15:29:38.822768) EVENT_LOG_v1 {"time_micros": 1732001378822755, "cf": "default", "job": 282, "event": "compaction_finished", "compaction_time_micros": 383849, "compaction_time_cpu_micros": 382345, "output_level": 2, "num_output_files": 2, "total_output_size": 130933454, "num_input_records": 128225, "num_output_records": 128225, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [5, 80, 92, 82, 0, 0, 0]} 2024/11/19-15:29:38.826945 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005257.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:38.826967 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001378826964, "job": 282, "event": "table_file_deletion", "file_number": 5257} 2024/11/19-15:29:38.830859 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005259.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:38.830880 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001378830877, "job": 282, "event": "table_file_deletion", "file_number": 5259} 2024/11/19-15:29:38.835049 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005262.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:38.835068 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001378835066, "job": 282, "event": "table_file_deletion", "file_number": 5262} 2024/11/19-15:29:38.839364 139851158521600 [delete_scheduler.cc:73] Deleted file /storage/mytopling/.rocksdb/005263.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024/11/19-15:29:38.839383 139851158521600 EVENT_LOG_v1 {"time_micros": 1732001378839381, "job": 282, "event": "table_file_deletion", "file_number": 5263} 2024/11/19-15:29:38.839515 139851158521600 [compaction_job.cc:2437] [default] [JOB 289] Compacting 4@2 files to L3, score 3.50, subcompactions 1 : 1 2024/11/19-15:29:38.839523 139851158521600 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 287 Base level 2, inputs: [5428(62MB) 5429(61MB) 5432(62MB) 5433(61MB)] 2024/11/19-15:29:38.984799 139851137550080 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00286/att-02 : response_code = 502 bad gateway 2024/11/19-15:29:38.984918 139851137550080 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00286/att-02/compact.done) = No such file or directory: 5.182 sec on http probe failed: 2, retry 2024/11/19-15:29:39.097096 139851169007360 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00288/att-01 : response_code = 502 bad gateway 2024/11/19-15:29:39.097212 139851169007360 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00288/att-01/compact.done) = No such file or directory: 7.186 sec on http probe failed: 3, retry 2024/11/19-15:29:39.163225 139851158521600 [ERROR] [dcompact_etcd.cc:1552] libcurl: https://aliyun-topling-saas.topling.cn:18443/dcompact : job-00289/att-00 : response_code = 500 : Connection refused (nlb-tnqh42046fsycuoovw.cn-shenzhen.nlb.aliyuncs.com:8000) 2024/11/19-15:29:39.164123 139851158521600 [ERROR] [dcompact_etcd.cc:1113] job-00289/att-00: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.324 sec, err: Corruption: rocksdb::Status rocksdb::DcompactEtcdExec::SubmitHttp(terark::fstring, const std::string&, size_t): Connection refused (nlb-tnqh42046fsycuoovw.cn-shenzhen.nlb.aliyuncs.com:8000), input = 254.474 MiB 2024/11/19-15:29:39.164313 139851158521600 [dcompact_etcd.cc:1773] job-00289/att-00: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:39.469462 139851158521600 [ERROR] [dcompact_etcd.cc:1552] libcurl: https://aliyun-topling-saas.topling.cn:18443/dcompact : job-00289/att-01 : response_code = 500 : Connection refused (nlb-tnqh42046fsycuoovw.cn-shenzhen.nlb.aliyuncs.com:8000) 2024/11/19-15:29:39.470342 139851158521600 [ERROR] [dcompact_etcd.cc:1113] job-00289/att-01: init time wrt = 0.000, mka = 0.000, mkw = 0.000, curl https://aliyun-topling-saas.topling.cn:18443/dcompact = 0.306 sec, err: Corruption: rocksdb::Status rocksdb::DcompactEtcdExec::SubmitHttp(terark::fstring, const std::string&, size_t): Connection refused (nlb-tnqh42046fsycuoovw.cn-shenzhen.nlb.aliyuncs.com:8000), input = 254.474 MiB 2024/11/19-15:29:39.470531 139851158521600 [dcompact_etcd.cc:1773] job-00289/att-01: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:39.593161 139851127064320 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00287/att-02 : response_code = 502 bad gateway 2024/11/19-15:29:39.593283 139851127064320 [dcompact_etcd.cc:1209] fopen(/storage/mytopling/.rocksdb/job-00287/att-02/compact.done) = No such file or directory: 5.179 sec on http probe failed: 2, retry 2024/11/19-15:29:39.707711 139850978166528 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00283/att-02 : response_code = 502 bad gateway 2024/11/19-15:29:39.707868 139850978166528 [dcompact_etcd.cc:1213] fopen(/storage/mytopling/.rocksdb/job-00283/att-02/compact.done) = No such file or directory: 11.205 sec on http probe failed: 5, giveup 2024/11/19-15:29:39.707894 139850978166528 [ERROR] [dcompact_etcd.cc:1318] DcompactEtcdExec::Attempt: job-00283/att-02: at http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/probe?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 estimate timeout 126.914 sec, real 11.490 (submit 276.572 ms wait 11.205 s), input 126.914 MiB, send shutdown 2024/11/19-15:29:39.712680 139850978166528 [ERROR] [dcompact_etcd.cc:1549] libcurl: http://cn-shenzhen.Aliyun.internal-saas-labour.topling.cn:8000/shutdown?labour=i-wz90axp17yqp1av7udge@10.0.3.178:8080 : job-00283/att-02 : response_code = 502 bad gateway 2024/11/19-15:29:39.712989 139850978166528 [dcompact_etcd.cc:1773] job-00283/att-02: CleanFiles 0.000 sec : 0.000 2024/11/19-15:29:39.715936 139850978166528 [dcompact_etcd.cc:288] remove_all(/storage/mytopling/.rocksdb/job-00283) = success 2024/11/19-15:29:39.715950 139850978166528 [dcompact_etcd.cc:1773] job-00283/att-03: CleanFiles 0.003 sec : 0.000 2024/11/19-15:29:39.715974 139850978166528 (Original Log Time 2024/11/19-15:29:05.220157) EVENT_LOG_v1 {"time_micros": 1732001345220150, "job": 283, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [5309, 5310, 5311, 5313], "score": 4.92863, "input_data_size": 134099136, "oldest_snapshot_seqno": 158448090} 2024/11/19-15:29:39.715984 139850978166528 [compaction_job.cc:2437] [default] [JOB 283] Compacting 4@1 files to L2, score 4.93, subcompactions 1 : 1 2024/11/19-15:29:39.715995 139850978166528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 281 Base level 1, inputs: [5309(31MB) 5310(31MB) 5311(31MB) 5313(31MB)] 2024/11/19-15:29:39.718043 139850978166528 [WARN] [top_zip_table_builder.cc:964] rocksdb::ToplingZipTableBuilder::ToplingZipTableBuilder(const rocksdb::ToplingZipTableFactory*, const rocksdb::TableBuilderOptions&, rocksdb::WritableFileWriter*): Exception: sideplugin/topling-sst/src/table/top_table_common.cc:55: void rocksdb::TempFileDeleteOnClose::open_temp(): errno=13 : ERROR: mkstemp(/storage/mytopling/..//tmp/Topling-82038-6r0GTe) = Permission denied