Post

ETW Internals: Architecture, Providers and Sessions

An introduction to Event Tracing for Windows: the controller, provider, session and consumer model, how providers describe their events through manifests, how sessions buffer and deliver them, and why ETW is the backbone of Windows security telemetry.

ETW Internals: Architecture, Providers and Sessions

Introduction

Event Tracing for Windows (ETW) is the operating system’s general-purpose, system-wide event tracing and logging mechanism. It was introduced in Windows 2000, and today almost every source of Windows telemetry is built on top of it: the Windows Event Log, performance tooling such as Windows Performance Recorder, Microsoft Defender and most third-party EDR products.

ETW is designed for high-volume, low-overhead tracing. Components emit events into in-memory buffers managed by the kernel, and the cost of an event that nobody is listening to is close to zero. Tracing can be switched on and off dynamically, without restarting the instrumented component.1

This article describes the ETW architecture, then looks at two of its building blocks, providers and sessions, through the tools a defender uses to inspect them.

Architecture

ETW defines four roles (Figure 1).1

flowchart TB
    CTRL["Controller"]

    subgraph ETW["ETW (kernel)"]
        direction LR
        S1["Session 1<br/>Buffer 1 … Buffer N"]
        S2["Session 2<br/>Buffer 1 … Buffer N"]
        SN["Session N<br/>Buffer 1 … Buffer N"]
    end

    PA["Provider A"]
    PB["Provider B"]
    PC["Provider C"]
    ETL[("ETL file")]
    C1["Consumer 1"]
    C2["Consumer 2"]

    CTRL -->|"start / stop sessions"| ETW
    CTRL -.->|"enable / disable"| PA
    PA -->|"events"| S1
    PB -->|"events"| S1
    PB -->|"events"| S2
    PC -->|"events"| SN
    SN -->|"real-time events"| C1
    S2 -->|"flush buffers"| ETL
    ETL -->|"logged events"| C2

    classDef controller fill:#ffd8a8,stroke:#e8590c,color:#000
    classDef provider fill:#a5d8ff,stroke:#1971c2,color:#000
    classDef session fill:#b2f2bb,stroke:#2f9e44,color:#000
    classDef consumer fill:#ffec99,stroke:#f08c00,color:#000
    class CTRL controller
    class PA,PB,PC provider
    class S1,S2,SN session
    class C1,C2 consumer

Figure 1. ETW architecture: controllers manage sessions, providers write events into session buffers, consumers read them in real time or from a log file.

Role Definition Typical API
Provider A kernel-mode or user-mode component that generates ETW events EventRegister, EventWrite
Session A kernel-managed object that aggregates the events of one or more providers into its own set of buffers. Buffers are scoped to the session (managed by the kernel)
Controller An application that creates, configures and stops sessions, and enables or disables providers for them StartTrace, EnableTraceEx2, ControlTrace
Consumer An application that subscribes to a session in real time, or opens a log file, and reads the events OpenTrace, ProcessTrace

The roles are decoupled, and that is the key property of the design:

  • A provider does not know who, if anyone, is listening. It only checks whether it is enabled and at which level and keywords.
  • A single provider can be enabled in several sessions at once, each with its own filters (Provider B in Figure 1).
  • A session can deliver events in real time to a consumer, to a log file (.etl) for later analysis, or both.2

Figure 2 shows the same interaction as a sequence of API calls.

sequenceDiagram
    autonumber
    participant Ctl as Controller
    participant K as ETW (kernel)
    participant P as Provider
    participant Con as Consumer

    P->>K: EventRegister(ProviderGuid)
    Ctl->>K: StartTrace(SessionName, Properties)
    Ctl->>K: EnableTraceEx2(Session, ProviderGuid, Level, Keywords)
    K-->>P: Enable callback (level, keywords)
    P->>K: EventWrite(EventDescriptor, Data)
    Note over K: Event stored in the session's buffers
    Con->>K: OpenTrace(SessionName) / ProcessTrace()
    K-->>Con: EventRecordCallback(EVENT_RECORD)
    Ctl->>K: ControlTrace(Session, EVENT_TRACE_CONTROL_STOP)

Figure 2. Life cycle of a trace, from the provider’s registration to the end of the session.

Providers

Each provider is identified by a GUID that uniquely identifies it across the system, and usually also by a name such as Microsoft-Windows-Kernel-Process. A provider describes the events it can emit in an instrumentation manifest, which defines for every event:3

  • an event ID and a version: the same ID can have several versions with different payloads;
  • a level (Critical, Error, Warning, Informational, Verbose);
  • one or more keywords: a 64-bit mask that groups events into categories a controller can enable selectively;
  • a task and an opcode (for example Start and Stop);
  • a template: the name and type of each field in the event payload.

Windows supports several provider technologies: manifest-based providers, classic MOF providers, WPP software tracing and TraceLogging. Manifest-based providers are the most common source of security telemetry, because their schema is published and can be decoded by any consumer.3

Example: Microsoft-Windows-Kernel-Process

Microsoft-Windows-Kernel-Process is one of the most valuable providers for a defender. Inspected with a manifest browser such as EtwExplorer,4 its summary reads:

Property Value
Provider name Microsoft-Windows-Kernel-Process
Provider GUID {22FB2CD6-0E7B-422B-A0C7-2FAD1FD0E716}
Provider symbol MicrosoftWindowsKernelProcess
Keywords 11
Events 40
Tasks 19
Templates 24

Its events cover the life cycle of processes, threads and images, plus priority changes. An excerpt:

Event ID Versions Keyword Opcode
ProcessStart 1 0–3 WINEVENT_KEYWORD_PROCESS Start
ProcessStop 2 0–2 WINEVENT_KEYWORD_PROCESS Stop
ThreadStart 3 0–1 WINEVENT_KEYWORD_THREAD Start
ThreadStop 4 0–1 WINEVENT_KEYWORD_THREAD Stop
ImageLoad 5 0 WINEVENT_KEYWORD_IMAGE Info
ImageUnload 6 0 WINEVENT_KEYWORD_IMAGE Info
CpuBasePriorityChange 7 0 WINEVENT_KEYWORD_CPU_PRIORITY Info
CpuPriorityChange 8 0 WINEVENT_KEYWORD_CPU_PRIORITY Info
PagePriorityChange 9 0 WINEVENT_KEYWORD_OTHER_PRIORITY Info
IoPriorityChange 10 0 WINEVENT_KEYWORD_OTHER_PRIORITY Info

The template of ProcessStart version 2 shows what a consumer receives for every new process:

Field Type
ProcessID UInt32
CreateTime FILETIME
ParentProcessID UInt32
SessionID UInt32
Flags UInt32
ImageName UnicodeString
ImageChecksum UInt32
TimeDateStamp UInt32
PackageFullName UnicodeString
PackageRelativeAppId UnicodeString

Because ProcessStart is gated by the WINEVENT_KEYWORD_PROCESS keyword, a controller interested only in process creation can enable that single keyword and avoid the much higher volume of thread and image events.

Experiment: inspecting a provider from the command line

No third-party tool is required to inspect a provider. logman lists its GUID, keywords and levels, and wevtutil dumps the full manifest, including events and templates:5

1
2
logman query providers Microsoft-Windows-Kernel-Process
wevtutil get-publisher Microsoft-Windows-Kernel-Process /getevents /getmessage

logman query providers with no arguments lists every provider registered on the system.

Sessions

A session is the unit ETW uses for configuration and buffering. Every session has a name, its own set of buffers, a delivery mode and a list of enabled providers with their filters. On a normal Windows installation dozens of sessions run permanently. Many are AutoLogger sessions, started at boot from configuration stored in the registry under HKLM\SYSTEM\CurrentControlSet\Control\WMI\Autologger.6

Example: the sessions of a running system

Performance Monitor shows the running sessions under Data Collector Sets → Event Trace Sessions. Each entry is a session. On a typical system the list includes, among others:

  • Circular Kernel Context Logger: an in-memory circular session for kernel events;
  • EventLog-Application, EventLog-System, EventLog-Security: the sessions that feed the Windows Event Log channels;
  • DiagLog, Diagtrack-Listener, WdiContextLog and many other diagnostic sessions.

The properties of a session map directly onto the concepts above. For DiagLog:

  • Trace Providers lists the providers enabled in the session, by name or by GUID (for example Microsoft-Windows-Application-Experience, Microsoft-Windows-AppXDeployment-Server, Microsoft-Windows-BootUX and Microsoft-Windows-COM), each with its filters:

    Property Value Meaning
    Keywords (Any) 0xFFFFFFFF Events matching any of these keywords
    Keywords (All) 0x0 Events must match all of these keywords
    Level 0xFF Events up to this level
    Properties 0x0000 Additional data to collect with each event
    Filter Disabled Optional event filter

    These fields correspond to the MatchAnyKeyword, MatchAllKeyword, Level and EnableProperty arguments of EnableTraceEx2.7

  • Trace Session holds the session name, the clock type used to timestamp events (performance counter, system time or CPU cycle counter) and the stream mode, which decides where events go.

  • Trace Buffers, File, Directory and Stop Condition control buffer sizes, the output log file and when the session stops.

The stream mode corresponds to the session’s logging mode:2

Stream mode Logging mode Delivery
Real Time EVENT_TRACE_REAL_TIME_MODE Events delivered to real-time consumers as buffers fill
File EVENT_TRACE_FILE_MODE_SEQUENTIAL or _CIRCULAR Events written to an .etl file
Real Time and File Both flags Both of the above
Buffering EVENT_TRACE_BUFFERING_MODE Events kept in memory only, in circular buffers

Experiment: listing sessions and their providers

logman shows the same information from the command line:5

1
2
3
4
5
# all running sessions (-ets targets live event trace sessions)
logman query -ets

# providers, keywords and level of a specific session
logman query "EventLog-System" -ets

Telemetry and hardening

ETW is the backbone of Windows telemetry, so its integrity is a security concern in its own right.

  • Visibility depends on sessions and providers. A detection that relies on an ETW provider works only while the session that consumes it is running and the provider is enabled in it. Stopping a session, disabling a provider or modifying an AutoLogger configuration silently removes that visibility. These actions are catalogued as defense evasion through indicator blocking (MITRE ATT&CK T1562.006).8
  • Monitor the configuration. Changes under HKLM\SYSTEM\CurrentControlSet\Control\WMI\Autologger (for example values such as Start or Enabled being set to 0) and unexpected invocations of session control tools (logman stop … -ets, logman update … -ets) deserve alerting.
  • Watch for gaps. As with AMSI, missing telemetry is itself a signal. A host that keeps running processes but stops producing process creation events from Microsoft-Windows-Kernel-Process needs to be investigated.
  • Mind the data. Some providers carry sensitive content, such as command lines, script text or network data. Real-time and file sessions should be configured with access control and retention policies in mind. Sessions have their own security descriptors, visible in the Security tab of the session properties.

Summary

ETW separates the act of emitting an event from the decision to collect it. Providers describe their events in manifests and write them cheaply. Controllers create sessions and enable providers in them with level and keyword filters. Sessions buffer events in kernel memory and deliver them in real time or to .etl files, where consumers read them. Tools such as EtwExplorer, Performance Monitor, logman and wevtutil expose every part of this model. Because so much of Windows security telemetry flows through it, the configuration of ETW sessions and providers is something a defender should both use and protect.

References

This post is licensed under CC BY 4.0 by the author.