Skip to content

feat(logging): add CerrLogger std::cerr backend (3/6)#724

Open
kamcheungting-db wants to merge 10 commits into
apache:mainfrom
kamcheungting-db:logging-block3-cerr
Open

feat(logging): add CerrLogger std::cerr backend (3/6)#724
kamcheungting-db wants to merge 10 commits into
apache:mainfrom
kamcheungting-db:logging-block3-cerr

Conversation

@kamcheungting-db

@kamcheungting-db kamcheungting-db commented Jun 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Part 3 of the logging stack (builds on #723). Adds the first concrete sink: a dependency-free std::cerr logger, which is also the default backend when spdlog is disabled.

What's here

  • CerrLogger writes one line per record: YYYY-MM-DDThh:mm:ss.mmmZ LEVEL [tid] file:line] message.
  • UTC millisecond timestamps, OS-native thread id (cached per thread), lock-free level check, and a mutex that guards the whole line so concurrent records never interleave.
  • Pure standard library — always available, no third-party dependency. Log/Flush never throw.

Tests — cerr_logger_test: line layout, level filtering, and concurrent-write safety. Built and run with clang/libc++.

This pull request and its description were written by Isaac.

@kamcheungting-db kamcheungting-db changed the title feat: [Iceberg Logger] [Part-3] CerrLogger std::cerr backend feat(logging): add CerrLogger std::cerr backend Jun 11, 2026
@kamcheungting-db kamcheungting-db changed the title feat(logging): add CerrLogger std::cerr backend feat(logging): add CerrLogger std::cerr backend (3/6) Jun 11, 2026
@kamcheungting-db kamcheungting-db force-pushed the logging-block3-cerr branch 6 times, most recently from 7f46cc5 to 0dbc548 Compare June 16, 2026 03:27
@kamcheungting-db kamcheungting-db mentioned this pull request Jun 16, 2026
Open
@kamcheungting-db
feat(logging): add Logger interface and process-global default (2/6)
246cab6 
Second block: the backend-agnostic logging API and the swappable default logger.

- Logger: pure-virtual sink (ShouldLog/Log/SetLevel/level/Flush/IsNoop), with
  ShouldLog() as the single authority for runtime filtering and a documented
  no-reentrancy / thread-safety / noexcept contract. Mirrors MetricsReporter.
- LogMessage owns its formatted text (moved in) so a sink may retain records;
  reserves an owning attributes vector for future structured logging without an
  ABI break. logger.h is backend-agnostic -- it never includes the build config
  header nor references the spdlog feature macro, so consumers see one stable API.
- Process-global default logger: GetDefaultLogger / SetDefaultLogger /
  SetDefaultLevel, plus a lock-free thread-local generation cache on the hot path
  (no lock or refcount churn in steady state; slot mutex only on swap/refresh).
  NoopLogger is the placeholder default here; CerrLogger and the spdlog backend
  arrive in the following blocks.

Builds the source into both CMake and Meson and installs logger.h alongside
log_level.h; logger_test covers the default-logger API, level lowering taking
effect immediately, and concurrent swap/read safety.

Co-authored-by: Isaac
@kamcheungting-db
fix(logging): address review feedback on the Logger interface
796ff0d 
Resolves @wgtmac's review comments on PR apache#723:

- Mark ShouldLog()/SetLevel()/level() noexcept so a custom logger cannot throw
  from the hot filter path; propagated to NoopLogger and CapturingLogger.
- Make CurrentLogger() safe during thread teardown without UB: the per-thread
  cache is reached via a trivially-destructible thread_local pointer to a
  never-freed heap slot (kept LSan-reachable by a registry), so logging from a
  thread_local destructor in any order is safe. Replaces the AliveFlag guard.
- Add ICEBERG_EXPORT to LogAttribute/LogMessage, matching Error/ScanReport.
- Trim the logger.h \file brief (the macros it referenced arrive in PR apache#725).
- Tests: use the IsError matcher and std::ignore; strengthen the teardown test
  to log from a thread_local destroyed after the cache.

Co-authored-by: Isaac
@kamcheungting-db
feat(logging): add std::format-style Log() functions
4943849 
Addresses @wgtmac's review comment on PR apache#723 asking for a std::format-style API
that builds the string only after the level check.

- A single overloaded iceberg::Log():
    Log(level, "fmt {}", args...)           -> process-default logger
    Log(logger, level, "fmt {}", args...)   -> explicit logger
  disambiguated by the first argument.
- Formats only after ShouldLog() (no wasted work when disabled) and never throws
  -- a formatting failure routes to EmitFormatError, matching the macros.
- The call-site source_location is captured via a small consteval FmtWithLoc
  wrapper (the std::print technique), preserving compile-time format-string
  checking without a macro.

The sink interface (Logger::Log(LogMessage&&)) intentionally stays
pre-formatted: passing std::format_args across the virtual boundary is
lifetime-unsafe and would break safe record retention. The format-style win is
delivered at the call layer instead.

Co-authored-by: Isaac
@kamcheungting-db
feat(logging): add LogMessage::Builder for structured attributes
bd4a25c 
Addresses @wgtmac's review comment on PR apache#723 that LogMessage::attributes had no
easy fill path.

Adds a fluent LogMessage::Builder so engine loggers can attach structured fields
(query id, task id, table name, snapshot id, file path, ...):

  auto record = LogMessage::Builder(LogLevel::kInfo)
                    .Message("scan finished")
                    .Attribute("table", table_name)
                    .Attribute("snapshot_id", std::to_string(id))
                    .Build();
  logger->Log(std::move(record));

- Message() sets the formatted text; Attribute() appends an owned key/value;
  Location() overrides the source location (defaults to the build site).
- Build() moves the accumulated state into a LogMessage.

This defines how attributes get populated, so structured logging can grow
without an ABI break.

Co-authored-by: Isaac
@kamcheungting-db
fix name problem
92c4f97
@kamcheungting-db
add comment to location_ to clarify why no std::move
5a9c820
@kamcheungting-db
fix(logging): free the per-thread logger cache at thread exit
79acfa8 
Replaces the intentional permanent per-thread leak with a cache that is freed at
thread exit, while staying safe to call CurrentLogger() from any thread_local
destructor during teardown.

The per-thread state is split by lifetime:
- `dead` (bool) and `cache` (pointer) are trivially destructible, so their
  storage stays valid for the whole thread and is readable from any other
  thread_local's destructor, in any destruction order.
- A `guard` (the only piece with a destructor) frees the cache and sets `dead`
  at thread exit. A thread_local destroyed after it reads `dead == true` and
  falls back to an immortal no-op; one destroyed before it still sees a live
  cache. Either order is safe.

This is the correct version of the earlier AliveFlag attempt, which was UB
because the flag object itself had a destructor (reading it after that
destructor ran was the very use-after-lifetime it tried to prevent). Drops the
LSan-reachability registry, which is no longer needed now that nothing leaks.

Validated with a dedicated stress harness under ThreadSanitizer and
AddressSanitizer (no leaks): logging from a thread_local destroyed after the
cache; logging interleaved with concurrent SetDefaultLogger swaps and thread
teardown; and a thread whose only logging happens in a thread_local destructor.
Adds ConcurrentTeardownAndSwapIsSafe to cover the combined path in CI.

Co-authored-by: Isaac
@kamcheungting-db
test(logging): make CapturingLogger's level atomic
963c49d
@kamcheungting-db
feat(logging): add ScopedLogger for per-context logger routing
642926c 
Bind a logger for the current thread + scope so the default path
(CurrentLogger/Log(level,...)/LOG_* macros) routes to it, with the global
default as fallback. Engines can route Iceberg's own internal logs into a
per catalog/session/query/task context without touching call sites.
@kamcheungting-db
feat(logging): add CerrLogger std::cerr backend (3/6)
895c800 
Third block: the first concrete sink, and the process default until the spdlog
backend lands.

- CerrLogger writes to std::cerr with a fixed line layout
  `YYYY-MM-DDThh:mm:ss.mmmZ LEVEL [tid] file:line] message`. Timestamps use UTC
  std::chrono floored to milliseconds (no gmtime/localtime -- thread-unsafe);
  the thread id is the OS-native id, cached per thread.
- Level is a std::atomic<LogLevel>; a mutex guards the whole formatted-line write
  so concurrent lines never interleave. Log()/Flush() wrap stream ops in
  try/catch so the noexcept contract holds even if the stream throws.
- MakeDefaultLogger() now returns CerrLogger.

Wired into both builds and installed; cerr_logger_test covers layout, level
filtering, and concurrent-write safety.

Co-authored-by: Isaac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kamcheungting-db