An Introduction to Media Foundation - Understanding the API Through a COM Lens

When you start working with Media Foundation, it is easy to feel that you were supposed to be using Windows’ video and audio APIs, and yet suddenly everything is about COM. CoInitializeEx, MFStartup, IMFSourceReader, IMFMediaType, IMFTransform, IMFActivate, HRESULT, and GUIDs all arrive at once, the atmosphere abruptly turns Win32/COM, and what Media Foundation actually is becomes hard to see.

Rather than covering all of Media Foundation like a dictionary, this article narrows down to three points.

• Why COM naturally comes up while you are using Media Foundation
• Where the COM flavor gets stronger
• Which of Source Reader / Sink Writer / Media Session / MFT to start with

The code examples are C++ based, but the thinking itself is essentially the same when you reach the API through a wrapper from .NET or a similar environment.

Table of Contents

1. The conclusion first, in one line
2. Vocabulary and the big picture
   • 2.1. Terms to grasp the meaning of first
   • 2.2. The big picture of Media Foundation (diagram)
3. Where Media Foundation puts on its COM face
   • 3.1. CoInitializeEx and MFStartup sit side by side at initialization
   • 3.2. Object hand-offs are interface-centric
   • 3.3. Settings and type information center on IMFAttributes and GUIDs
   • 3.4. Activation objects appear
   • 3.5. Asynchrony, callbacks, and threads are handled the COM way too
4. But Media Foundation is not the same as COM
5. Where to start (choosing an entry point)
   • 5.1. Cases where you start with the Source Reader
   • 5.2. Sink Writer if you are writing to a file
   • 5.3. Media Session if you handle playback and synchronization
   • 5.4. MFT if you are plugging in your own component
6. A checklist for real projects
7. Summary
8. References

Knowledge map for this article

This article explains that Media Foundation is a media processing platform built on top of COM, looking at it from five points: initialization, passing objects around, configuration, enumeration, and asynchronous processing. Initializing the COM library (CoInitializeEx) is a prerequisite for initializing Media Foundation itself (MFStartup), and components such as IMFSourceReader, IMFAttributes, and IMFTransform are all COM interfaces derived from IUnknown and return an HRESULT. IMFActivate is an entry point for creating the real object later, and an IMFTransform is obtained only after calling ActivateObject on an entry from the MFTEnumEx enumeration results. Asynchronous processing is called from work queue threads that run in the MTA, so the design has to avoid touching STA-side UI objects directly and instead hand only the results across.

flowchart LR
    accTitle: The relationship between Media Foundation and COM
    accDescr: Diagram showing that Media Foundation is a media processing platform built on COM, that the character of COM surfaces at each point of initialization, object representation, configuration, enumeration, and asynchronous processing, and that the MTA work queue has to be bridged to the apartment model of the UI
    media_foundation["Media Foundation"]
    com["COM (Component Object Model)"]
    mfstartup["MFStartup"]
    coinitializeex["CoInitializeEx"]
    iunknown["IUnknown"]
    hresult["HRESULT"]
    imfsourcereader["IMFSourceReader (Source Reader)"]
    imftransform["IMFTransform (MFT)"]
    imfattributes["IMFAttributes"]
    imfactivate["IMFActivate (Activation Object)"]
    imfsinkwriter["IMFSinkWriter (Sink Writer)"]
    media_session["Media Session"]
    mf_topology["Topology (Media Foundation)"]
    imfsourcereadercallback["IMFSourceReaderCallback"]
    mf_work_queue["Media Foundation Work Queue"]
    com_apartment_model["COM apartment model (STA/MTA)"]
    com_smart_pointer["COM smart pointer (ComPtr/wil::com_ptr)"]
    media_type_negotiation["Media Type Negotiation"]
    media_foundation -->|"uses"| com
    mfstartup -->|"requires"| coinitializeex
    media_foundation -->|"uses"| mfstartup
    com -->|"requires"| iunknown
    com -->|"uses"| hresult
    imfsourcereader -->|"uses"| com
    imftransform -->|"uses"| com
    imfattributes -->|"uses"| com
    imfactivate -->|"uses"| imfattributes
    media_foundation -.->|"uses"| imfsourcereader
    media_foundation -.->|"uses"| imfsinkwriter
    media_foundation -.->|"uses"| media_session
    media_session -->|"uses"| mf_topology
    imfsourcereader -->|"uses"| imftransform
    imfsourcereader -.->|"uses"| imfsourcereadercallback
    imfsourcereadercallback -->|"uses"| mf_work_queue
    mf_work_queue -->|"uses"| com_apartment_model
    com_smart_pointer -->|"recommended for"| com
    imfsourcereader -->|"requires"| media_type_negotiation
    media_type_negotiation -->|"uses"| imfattributes
    imftransform -.->|"requires"| imfactivate

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (21 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

1. The Conclusion First, in One Line

• Media Foundation is a platform for handling video and audio, and the API as a whole is not simply pure COM
• However, the boundaries of source / transform / sink / activation / attributes / callback are expressed as COM interfaces, so IUnknown, HRESULT, GUIDs, and apartments naturally come up while you use it
• It is easier to keep things straight if you start with the Source Reader or Sink Writer, move to the Media Session once you need playback control, and move to MFTs once you need a custom transform

In short, Media Foundation is a media processing platform, and COM is deeply embedded in its boundary surfaces.

Getting that straight first makes it far easier to see why the API suddenly puts on a COM face.

flowchart TB
    accTitle: The relationship between Media Foundation and COM
    accDescr: Shows that Media Foundation itself is a media processing platform and that the API as a whole is not pure COM, but because the boundaries between components are expressed as COM interfaces, IUnknown and HRESULT and GUID naturally come up.
    mf["Media Foundation"] --> plat["A media processing platform"]
    mf --> border["Component boundaries are COM interfaces"]
    border --> com["IUnknown, HRESULT and GUID show up"]
    plat -.-> note["The API as a whole is not pure COM"]

Figure 1: The core is a media processing platform, with COM deeply embedded in its boundary surfaces.

2. Vocabulary and the Big Picture

Before getting into COM, let us go over the vocabulary this article uses and the broad shape of Media Foundation.

2.1. Terms to Grasp the Meaning of First

Having this vocabulary in hand first removes a lot of the friction when you read the documentation.

Apartments become the main topic in 3.5, but here it is enough to remember that a Media Foundation callback can arrive on a thread other than the one you called ReadSample on, namely a thread in the MTA work queue.

2.2. The Big Picture of Media Foundation (Diagram)

Seen from a distance, Media Foundation is about media pipelines. COM matters, but it is easier to organize things if you look at the big picture first.

flowchart TB
    subgraph Pipeline["Model that uses the whole pipeline"]
        Source1["Media Source"] --> Transform1["MFT"]
        Transform1 --> Sink1["Media Sink"]
        Session["Media Session"] --- Source1
        Session --- Transform1
        Session --- Sink1
    end

    subgraph Direct["Model where the app handles data directly"]
        Source2["Media Source"] --> Reader["Source Reader (+ decoder)"]
        Reader --> App["App"]
        App --> Writer["Sink Writer (+ encoder)"]
        Writer --> Sink2["Media Sink"]
    end

Figure 2: Two ways to use it. One leaves the pipeline to the Media Session; the other has the app handle data directly through the Reader and Writer.

Broadly speaking, there are two ways to use Media Foundation.

• The model that uses the whole pipeline
  • You connect source / transform / sink, and the Media Session manages the data flow and A/V synchronization
• The model where the app handles the data directly
  • You pull data out of a source with the Source Reader and push it into a sink with the Sink Writer

The latter is the easier way in when you want to process frames or samples yourself. The former is the main path when you want to leave playback and synchronization to the platform as well.

The point to hold onto is that Media Foundation is really a media processing platform, which feels a little different from handling a loose collection of COM objects directly.

That said, once you start looking at the boundaries between its components, the COM face suddenly gets much stronger. The next chapter walks through those points in order.

3. Where Media Foundation Puts On Its COM Face

The places where the COM flavor gets stronger fall into roughly the following five points.

Only the last one, playback control, is a different animal: it is not general COM but Media Foundation’s own functionality. That is why chapter 4 covers it separately.

Below we look at them in order. The code is not a complete sample, only excerpts long enough to show where the COM face appears.

3.1. CoInitializeEx and MFStartup Sit Side by Side at Initialization

This is where most people first feel that something is off. Before you get to opening a file or capturing from a camera, CoInitializeEx and MFStartup show up.

• CoInitializeEx initializes the COM library
• MFStartup initializes the Media Foundation platform

In other words, COM initialization alone is not enough; Media Foundation needs its own initialization too. This is the point where you start to see that this is not just a video API, and that there is quite a lot of COM based contract underneath.

template <class T>
void SafeRelease(T** pp)
{
    if (pp != nullptr && *pp != nullptr)
    {
        (*pp)->Release();
        *pp = nullptr;
    }
}

HRESULT InitializeMediaFoundationForCurrentThread()
{
    HRESULT hr = CoInitializeEx(nullptr, COINIT_MULTITHREADED);
    if (FAILED(hr))
    {
        return hr;
    }

    hr = MFStartup(MF_VERSION);
    if (FAILED(hr))
    {
        CoUninitialize();
        return hr;
    }

    return S_OK;
}

void UninitializeMediaFoundationForCurrentThread()
{
    MFShutdown();
    CoUninitialize();
}

This pairing of CoInitializeEx and MFStartup is the first place where the COM air suddenly gets thicker while you are working with Media Foundation.

flowchart TB
    accTitle: The two stages of initialization and shutdown
    accDescr: Shows the two-stage arrangement where CoInitializeEx initializes the COM library and MFStartup initializes the Media Foundation platform before use, and shutdown calls MFShutdown and CoUninitialize in the reverse order.
    co["CoInitializeEx (COM initialization)"] --> mfs["MFStartup (MF initialization)"]
    mfs --> use["Use Media Foundation"]
    use --> shut["MFShutdown"]
    shut --> coun["CoUninitialize"]

Figure 3: COM initialization alone is not enough. Initialization comes in two stages, and shutdown unwinds them in reverse.

On real projects, deciding the following at this point makes life easier later.

• Which thread uses Media Foundation
• Whether that thread is STA or MTA
• Who owns MFStartup / MFShutdown and CoInitializeEx / CoUninitialize

In an actual implementation, another layer may already be responsible for COM initialization. Even then, it is safer to fix who owns that responsibility up front. If you move ahead with this design left vague, callbacks and UI integration become hard to reason about later.

Note that the code in this article manages raw interface pointers with a hand-written SafeRelease. The samples in Microsoft’s documentation are written this way, and it makes it visible where AddRef / Release take effect. That is not a reason to avoid smart pointers in production code, though. If you are writing new C++, it is safer to settle on one of the following.

With ComPtr, the combination of goto done; and SafeRelease shown later in 3.1 becomes unnecessary, because Release happens as soon as the scope exits. This article keeps raw pointers so that the COM conventions stay visible, but new code should start from ComPtr.

flowchart TB
    accTitle: Choosing between raw pointers and smart pointers
    accDescr: Shows that the code in this article uses raw pointers and SafeRelease so the effect of AddRef and Release stays visible, while new production code is safer with ComPtr or wil::com_ptr because Release happens as soon as the scope exits.
    raw["Raw pointers and SafeRelease"] -.-> why["A style that shows where Release takes effect"]
    smart["ComPtr or wil::com_ptr"] --> auto["Release happens when the scope exits"]
    auto --> rec["Start new code from ComPtr"]

Figure 4: The code in the article stays on raw pointers for learning purposes. New production code should settle on smart pointers.

3.2. Object Hand-Offs Are Interface-Centric

As you read through the Media Foundation API, most return values and out parameters are COM interfaces.

• IMFSourceReader
• IMFMediaType
• IMFTransform
• IMFActivate
• IMFSample
• IMFMediaBuffer

What stands out is that not only the data itself but also type information and configuration objects are expressed as interfaces.

For example,

• IMFTransform is the interface that represents an MFT
• IMFAttributes is a key/value store
• IMFMediaType is a description of a media format that inherits from IMFAttributes

So even something that looks like configuration data, such as a media type, is held as a COM interface. This is where IUnknown, QueryInterface, AddRef / Release, and HRESULT naturally enter the picture.

flowchart TD
    IUnknown["IUnknown"]
    IUnknown --> IMFAttributes["IMFAttributes"]
    IMFAttributes --> IMFMediaType["IMFMediaType"]
    IMFAttributes --> IMFActivate["IMFActivate"]
    IUnknown --> IMFSourceReader["IMFSourceReader"]
    IUnknown --> IMFTransform["IMFTransform"]

Figure 5: The lineage of the main interfaces. Even settings and type information are expressed as COM interfaces rooted in IUnknown.

By this point you can see that Media Foundation is a media API, but that the way it expresses boundaries is quite COM-like.

3.3. Settings and Type Information Center on IMFAttributes and GUIDs

There is a point in working with Media Foundation where the settings suddenly look like nothing but GUIDs. IMFAttributes is at the center of that: a key/value store keyed by GUID. It is used very heavily throughout Media Foundation.

IMFMediaType matters especially. It inherits from IMFAttributes and holds information about a media format as attributes.

For example, information like this.

• major type (audio or video)
• subtype (H.264, AAC, RGB32, PCM, and so on)
• frame size
• frame rate
• sample rate
• channel count

flowchart LR
    MediaType["IMFMediaType"] --> Major["MF_MT_MAJOR_TYPE"]
    MediaType --> Subtype["MF_MT_SUBTYPE"]
    MediaType --> Detail["Size / FPS / sample rate and so on"]

Figure 6: IMFMediaType is an attribute store that holds format information such as the major type and subtype under GUID keys.

It is easy to experience this as a forest of GUIDs, but what is actually going on is fairly straightforward.

• Hold settings in an attribute store
• Express media types as attribute stores as well
• Agree on a format between source / transform / sink by looking at those attributes

It is simply that COM style interfaces and GUIDs are used to express settings and type information.

Putting all of this into code looks like the following. It is an example that only reads one frame from a video with the Source Reader.

HRESULT ReadOneVideoSample(PCWSTR path)
{
    IMFSourceReader* pReader = nullptr;
    IMFMediaType* pType = nullptr;
    IMFSample* pSample = nullptr;

    HRESULT hr = MFCreateSourceReaderFromURL(path, nullptr, &pReader);
    if (FAILED(hr)) goto done;

    hr = MFCreateMediaType(&pType);
    if (FAILED(hr)) goto done;

    hr = pType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
    if (FAILED(hr)) goto done;

    hr = pType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_RGB32);
    if (FAILED(hr)) goto done;

    hr = pReader->SetCurrentMediaType(
        MF_SOURCE_READER_FIRST_VIDEO_STREAM,
        nullptr,
        pType);
    if (FAILED(hr)) goto done;

    DWORD streamFlags = 0;
    LONGLONG timestamp = 0;

    hr = pReader->ReadSample(
        MF_SOURCE_READER_FIRST_VIDEO_STREAM,
        0,
        nullptr,
        &streamFlags,
        &timestamp,
        &pSample);
    if (FAILED(hr)) goto done;

    // Get an IMFMediaBuffer out of pSample and process it

done:
    SafeRelease(&pSample);
    SafeRelease(&pType);
    SafeRelease(&pReader);
    return hr;
}

What becomes visible here is the following.

• Both the reader and the media type are COM interfaces
• Settings are GUID based
• The return value is an HRESULT
• In synchronous mode, ReadSample blocks

Even when all you want is to read a single frame, the Media Foundation boundary puts on a fairly COM-like face. The last point about synchronous mode is covered in 3.5.

The steps of media type negotiation (this is one of the three things listed as worth looking at first in the chapter 6 checklist)

The code above only declares that it wants RGB32, so real work needs steps before and after it. For the Source Reader, the flow Microsoft’s documentation lays out has the following four steps.

1. Enumerate the native types - Call IMFSourceReader::GetNativeMediaType(streamIndex, typeIndex, &pType) with typeIndex starting at 0 and increasing. Once you go past the range it returns MF_E_NO_MORE_TYPES, which marks the end of the enumeration (if streamIndex is out of range, you get MF_E_INVALIDSTREAMNUMBER). A file often has just one type per stream, but a webcam has several formats
2. Check the major type - Read MF_MT_MAJOR_TYPE from the media type you enumerated and decide whether it is audio or video. If you skip this and hard-code an assumption instead, you will end up throwing video settings at an audio stream
3. Build and set the output format you want - Create a new media type with MFCreateMediaType, set MF_MT_MAJOR_TYPE and MF_MT_SUBTYPE, and call SetCurrentMediaType. If you want to receive the data still compressed, pass the type you got in step 1 as is; if you want it decoded, specify an uncompressed format such as MFVideoFormat_RGB32 or MFAudioFormat_PCM. The Source Reader loads the decoder for you
4. Read back the format that was settled on - After SetCurrentMediaType, call GetCurrentMediaType to get the details of the format that was actually settled on (frame size, stride, sample rate, and so on). What you pass in step 3 is only a partial specification, so reading the final values here is the correct order

If you skip these four steps and proceed on the assumption that it is probably this format, you either get MF_E_INVALIDMEDIATYPE back or, even if the call goes through, you read buffers in a format you did not expect.

flowchart TB
    accTitle: The four steps of media type negotiation
    accDescr: Shows the four steps of enumerating native types with GetNativeMediaType, checking the major type, building the desired output format and applying it with SetCurrentMediaType, and finally reading back the settled format with GetCurrentMediaType.
    s1["Enumerate with GetNativeMediaType"] --> s2["Check the major type"]
    s2 --> s3["Apply the format you want with SetCurrentMediaType"]
    s3 --> s4["Read the settled values with GetCurrentMediaType"]
    s1 -.-> stop["MF_E_NO_MORE_TYPES marks the end of enumeration"]

Figure 7: Format negotiation takes four steps. What you pass in is a partial specification, so read the settled values back at the end.

3.4. Activation Objects Appear

Activation objects are where the COM flavor of Media Foundation shows up most.

IMFActivate is a helper object for creating the real object later. It is easiest to picture it as something close to a COM class factory.

Where these appear, the return value of an enumeration API is not the real object you can use as is, but first an array of IMFActivate*. You then instantiate only the ones you need with ActivateObject.

sequenceDiagram
    participant App as App
    participant Enum as Enumeration API
    participant Act as IMFActivate
    participant Obj as IMFTransform / Sink and so on

    App->>Enum: Call the enumeration
    Enum-->>App: Array of IMFActivate*
    App->>Act: Check the attributes
    App->>Act: ActivateObject(...)
    Act-->>App: The real COM object

Figure 8: What the enumeration API returns is an IMFActivate, and only after calling ActivateObject do you get the real COM object.

This shape fits well with the fact that Media Foundation is designed to discover replaceable components later and combine them.

Also, since an activation object can hold attributes of its own, the flow tends to be to look at a candidate’s attributes first, set something on it if needed, and instantiate it later. This too is quite COM-like.

Actually enumerating an MFT with MFTEnumEx and instantiating it looks like this.

HRESULT FindH264Decoder(IMFTransform** ppTransform)
{
    *ppTransform = nullptr;

    IMFActivate** ppActivate = nullptr;
    UINT32 count = 0;

    MFT_REGISTER_TYPE_INFO inputType = {};
    inputType.guidMajorType = MFMediaType_Video;
    inputType.guidSubtype = MFVideoFormat_H264;

    HRESULT hr = MFTEnumEx(
        MFT_CATEGORY_VIDEO_DECODER,
        MFT_ENUM_FLAG_SYNCMFT | MFT_ENUM_FLAG_LOCALMFT,
        &inputType,
        nullptr,
        &ppActivate,
        &count);
    if (FAILED(hr))
    {
        return hr;
    }

    if (count == 0)
    {
        CoTaskMemFree(ppActivate);
        return MF_E_TOPO_CODEC_NOT_FOUND;
    }

    hr = ppActivate[0]->ActivateObject(
        __uuidof(IMFTransform),
        reinterpret_cast<void**>(ppTransform));

    for (UINT32 i = 0; i < count; ++i)
    {
        ppActivate[i]->Release();
    }
    CoTaskMemFree(ppActivate);

    return hr;
}

The enumeration result is not IMFTransform* from the start; it comes back as IMFActivate**, and only when you call ActivateObject do you finally get the real IMFTransform. This flow captures the sense that Media Foundation suddenly puts on a COM face rather well.

flowchart TB
    accTitle: The flow from MFTEnumEx to instantiating a decoder
    accDescr: Shows the flow where MFTEnumEx enumerates decoder candidates and returns an array of IMFActivate, where no candidate means MF_E_TOPO_CODEC_NOT_FOUND, where a candidate is instantiated with ActivateObject to obtain an IMFTransform, and where each IMFActivate is released and the array itself is freed with CoTaskMemFree.
    enum["Enumerate candidates with MFTEnumEx"] --> arr["An array of IMFActivate comes back"]
    arr --> q{"Are there candidates"}
    q -->|"No"| nf["MF_E_TOPO_CODEC_NOT_FOUND"]
    q -->|"Yes"| act["Instantiate with ActivateObject"]
    act --> obj["Get an IMFTransform"]
    act -.-> free["Release each element"]
    free -.-> free2["Free the array with CoTaskMemFree"]

Figure 9: Enumerate, check candidates, instantiate, release. The enumeration result is not the real object you can use as is.

3.5. Asynchrony, Callbacks, and Threads Are Handled the COM Way Too

Asynchronous processing and the threading model are easy to overlook in practical Media Foundation work.

The Source Reader, for example, is in synchronous mode by default. In synchronous mode, ReadSample blocks. Depending on the state of the file, the network, or the device, that wait can become long enough to notice.

To use asynchronous mode, you pass a callback when you create the Source Reader. The flow is to prepare an object that implements IMFSourceReaderCallback, set it on the MF_SOURCE_READER_ASYNC_CALLBACK attribute, and then create the reader.

HRESULT CreateSourceReaderAsync(
    PCWSTR path,
    IMFSourceReaderCallback* pCallback,
    IMFSourceReader** ppReader)
{
    IMFAttributes* pAttributes = nullptr;

    HRESULT hr = MFCreateAttributes(&pAttributes, 1);
    if (FAILED(hr))
    {
        return hr;
    }

    hr = pAttributes->SetUnknown(MF_SOURCE_READER_ASYNC_CALLBACK, pCallback);
    if (SUCCEEDED(hr))
    {
        hr = MFCreateSourceReaderFromURL(path, pAttributes, ppReader);
    }

    SafeRelease(&pAttributes);
    return hr;
}

In other words,

• the callback itself is a COM interface
• the asynchronous setting goes through IMFAttributes
• the mode is decided at creation time

That is the shape of it.

Apartments matter somewhat more here. Media Foundation’s asynchronous processing uses work queues, and work queue threads are MTA. That is why the implementation becomes simpler if the application side also works in the MTA.

sequenceDiagram
    participant App as App thread
    participant Reader as Source Reader
    participant Queue as MF work queue (MTA)
    participant Cb as IMFSourceReaderCallback

    App->>Reader: ReadSample(...)
    Reader-->>App: Returns immediately
    Reader->>Queue: Processes internally
    Queue->>Cb: OnReadSample(...)

Figure 10: In asynchronous mode ReadSample returns immediately, and OnReadSample is called from a work queue thread.

What to watch out for around callbacks is this.

• Do not touch STA objects owned by the UI thread directly from the callback
• Make the callback implementation thread-safe
• If you need to update the UI, send only the result back to the UI thread
• Decide up front which thread Media Foundation callbacks arrive on

Media Foundation does not quietly absorb the constraints of STA objects for you. That is why it is easier to keep things straight if you keep the workers that use Media Foundation in the MTA and bridge to the UI explicitly.

flowchart TB
    accTitle: How to bridge callbacks and the UI thread
    accDescr: Shows the arrangement where callbacks arrive from MTA work queue threads so the implementation is made thread-safe, STA UI objects are not touched directly, and only the result is sent back to the UI thread when an update is needed.
    cb["Callbacks arrive from the MTA work queue"] --> safe["Make the implementation thread-safe"]
    cb -.-> ng["Do not touch STA UI objects directly"]
    safe --> bridge["Send only the result back to the UI thread"]

Figure 11: Keep the workers that use Media Foundation in the MTA and bridge to the UI explicitly.

4. But Media Foundation Is Not the Same as COM

Reading this far, it is easy to conclude that Media Foundation is really just COM. That is not quite right, though.

Media Foundation has platform specific concepts that general COM does not account for.

• MFStartup / MFShutdown
• Media Session
• topology
• topology loader
• presentation clock
• Source Reader / Sink Writer

These are Media Foundation’s own role: how media flows through the pipeline.

With the Media Session, for example, the application hands over a partial topology, and the topology loader fills in the transforms that are needed to resolve it into a full topology. That is not a general COM matter but functionality Media Foundation has as a media processing platform.

flowchart LR
    Partial["Partial Topology<br/>Source -> Output"] --> Loader["Topology Loader"]
    Loader --> Full["Full Topology<br/>Source -> Decoder MFT -> Output"]

Figure 12: Hand over a partial topology and the topology loader fills in the transforms needed to resolve it into a full topology.

Media Foundation uses COM to express the contracts between components, and on top of that runs as a media processing platform. Looking at it as these two layers keeps you from getting lost.

flowchart TB
    accTitle: The two layers of COM and the platform
    accDescr: Shows that Media Foundation expresses the contracts between components with COM and on top of that has pipeline specific machinery such as the Media Session and topologies and the presentation clock, giving it a two-layer structure.
    com2["The COM layer (expresses component contracts)"] --> mf2["The media processing platform layer"]
    mf2 --> own["Media Session, topology and so on"]
    own -.-> note2["MF specific concepts that general COM does not cover"]

Figure 13: Not a rehash of COM. On top of the COM layer sits an MF specific layer that moves the pipeline.

5. Where to Start (Choosing an Entry Point)

When you are deciding on your first entry point, the following diagram is often enough.

flowchart TD
    Start["What you want to do"] --> Q1{"What do you need first?"}
    Q1 -- "Read frames / samples" --> A1["Source Reader"]
    Q1 -- "Write to a file" --> A2["Sink Writer"]
    Q1 -- "Playback control or A/V sync" --> A3["Media Session"]
    Q1 -- "Insert a custom transform" --> A4["MFT"]

Figure 14: Choose the entry point from what you need first. Reader to read, Writer to write, Session to play back.

As a table, it looks like this.

5.1. Cases Where You Start With the Source Reader

The Source Reader is quite easy to use as the entry point when you want to pull data out of a file or a device.

It suits cases like these.

• You want frames from a video file
• You want to decode an audio file and get samples
• You want frames from a camera
• You want to connect a Media Foundation source to your own processing pipeline

The Source Reader loads a decoder as needed and hands the data to the application. On the other hand, it does not take care of managing the presentation clock, A/V synchronization, or the rendering to the screen itself.

It is easiest to think of it as an entry point for getting data, not for playing it back.

flowchart TB
    accTitle: The scope of the Source Reader
    accDescr: Shows the scope where the Source Reader pulls data out of files and cameras and loads a decoder as needed to hand data to the app, but does not take care of managing the presentation clock, A/V synchronization, or rendering to the screen.
    src["Sources such as files and cameras"] --> sr["Source Reader"]
    sr --> app["Hand data to the app"]
    sr -.-> dec["Load a decoder if needed"]
    sr -.-> not["Does not take care of playback or synchronization"]

Figure 15: The Source Reader is an entry point for getting data, not for playing it back.

5.2. Sink Writer if You Are Writing to a File

The Sink Writer is the entry point when you want to write audio or video out to a file.

Typical uses are around here.

• You want to save generated frames to a video file
• You want to encode audio samples and write them out
• You want to convert data you read into another format and save it

The Sink Writer finds and loads an encoder as needed and manages the data flow to the media sink. It is often combined with the Source Reader, but the two are independent components, so you do not have to use them as a set.

flowchart TB
    accTitle: The scope of the Sink Writer
    accDescr: Shows that when the app hands generated frames and audio samples to the Sink Writer, it finds and loads an encoder as needed and manages the data flow to the media sink so the data is written out to a file.
    app2["Frames and audio the app produced"] --> sw["Sink Writer"]
    sw -.-> enc["Load an encoder if needed"]
    sw --> sink["Write out to a media sink"]
    sw -.-> ind["A component independent of the Source Reader"]

Figure 16: The Sink Writer is the entry point that takes care of encoding and writing out. Pairing it with the Reader is not required.

5.3. Media Session if You Handle Playback and Synchronization

If what you want is not to get data from a file but to play it back properly, it is more natural to center your design on the Media Session.

The Media Session comes into play when you have requirements like these.

• You want to handle play / stop / seek
• You want the platform to take care of synchronizing audio and video
• You want to handle the pipeline including quality control and format changes
• You want to build the source / transform / sink flow with a topology

Once you enter this layer, you move closer to Media Foundation proper than the Source Reader or Sink Writer do. That also brings in more Media Foundation specific concepts, such as topologies and session events.

flowchart TB
    accTitle: Deciding to use the Media Session
    accDescr: Shows that when you want the platform to take care of playback and stop and seek as well as A/V synchronization and quality control, you center the design on the Media Session and build the flow with a topology, which brings in more Media Foundation specific concepts.
    need["Want playback, seeking and sync handled for you"] --> ms["Center the design on the Media Session"]
    ms --> topo["Build source to sink with a topology"]
    ms -.-> deep["More MF specific concepts come with it"]

Figure 17: If the goal is proper playback rather than just getting data, the Media Session is the main path.

5.4. MFT if You Are Plugging In Your Own Component

An MFT is Media Foundation’s common model for transforms.

You come here in situations like these.

• You want to write your own decoder or encoder
• You want to plug a video or audio processing component into the pipeline
• You want to enumerate codecs and transforms and choose one yourself
• You want deeper control than the default automatic resolution

In the world of MFTs, COM style contracts come well into the foreground: IMFTransform, IMFActivate, media type negotiation, sample and buffer management, and so on. That is why it is clearer to first see which of the Source Reader, Sink Writer, or Media Session you actually need, rather than jumping straight into MFTs as your first entry point.

flowchart TB
    accTitle: What to check before moving on to MFTs
    accDescr: Shows that you move to MFTs when you want to plug your own decoder or transform into the pipeline, but because COM style contracts come to the foreground it is better to first check whether the Source Reader or Sink Writer or Media Session is enough.
    first["See whether the other three entry points are enough"] --> q{"Do you need a custom transform"}
    q -->|"No"| use3["Continue with the Reader, Writer or Session"]
    q -->|"Yes"| mft["Move on to MFT (IMFTransform)"]
    mft -.-> heavy["COM style contracts come to the foreground"]

Figure 18: The MFT is the last entry point. Do not jump straight in; confirm the other three are not enough first.

6. A Checklist for Real Projects

Finally, here is a single page of the things worth looking at first on real projects.

Three of these have the highest priority.

1. Do not pick the wrong entry point API
   • First work out which of the Source Reader, Sink Writer, or Media Session you actually need
2. Decide the apartment first
   • If you are going to mix an STA UI with Media Foundation work queues, decide how to bridge them at the start
3. Do not be sloppy about media type negotiation
   • Proceeding on the assumption that it is probably this format makes things much harder to follow later
   • The concrete procedure is collected in “The steps of media type negotiation” in 3.3

flowchart TB
    accTitle: The three highest priority items in the checklist
    accDescr: Shows that not picking the wrong entry point API, deciding the apartment first, and not being sloppy about media type negotiation are the three highest priority items, and that they let you avoid confusion later in the implementation.
    c1["Do not pick the wrong entry point API"] --> ease["Avoid confusion later in the implementation"]
    c2["Decide the apartment first"] --> ease
    c3["Do not be sloppy about format negotiation"] --> ease

Figure 19: Of everything in the checklist, nailing these three down first is what pays off.

7. Summary

It is no accident that COM comes up so much once you start working with Media Foundation.

• Media Foundation is a media processing platform
• Its boundaries, such as source / transform / sink / activation / callback, are expressed as COM interfaces
• Because of that, IUnknown, HRESULT, GUIDs, apartments, and callbacks naturally come up
• Media Foundation itself, however, is a media pipeline with a Media Session and topologies, not simply COM rewritten

On real projects, thinking in the following order keeps things quite clear.

1. Work out which of the Source Reader, Sink Writer, Media Session, or MFT you need in the first place
2. Decide the apartment and callback policy first
3. Handle media type negotiation and object lifetimes carefully

flowchart TB
    accTitle: The order to think in on real projects
    accDescr: Shows the order for sorting things out on real projects, where you first work out which entry point you need, then decide the apartment and callback policy, and finally handle media type negotiation and object lifetimes carefully.
    o1["Work out which entry point you need"] --> o2["Decide the apartment and callback policy"]
    o2 --> o3["Handle format negotiation and lifetimes carefully"]

Figure 20: The order to think in on real projects. Pick the entry point, set the threading policy, then settle formats and lifetimes.

You do not have to understand all of it from the start. Seeing it first as a media processing platform, with COM deeply embedded in its boundary surfaces, makes both the documentation and the code much easier to follow.

8. References

• Media Foundation and COM - Microsoft Learn
• Overview of the Media Foundation Architecture - Microsoft Learn
• Initializing Media Foundation - Microsoft Learn
• Source Reader - Microsoft Learn
• Using the Source Reader to Process Media Data - Microsoft Learn
• Using the Source Reader in Asynchronous Mode - Microsoft Learn
• Sink Writer - Microsoft Learn
• Activation Objects - Microsoft Learn
• About Topologies - Microsoft Learn
• IMFAttributes interface - Microsoft Learn
• IMFMediaType interface - Microsoft Learn
• IMFTransform interface - Microsoft Learn
• MFTEnumEx function - Microsoft Learn
• IMFSourceReader::GetNativeMediaType - Microsoft Learn
• ComPtr Class (Microsoft::WRL) - Microsoft Learn
• COM STA/MTA Fundamentals - Threading Models and How to Avoid Hangs | KomuraSoft Blog
• Calling Native DLLs from C#: C++/CLI Wrapper vs P/Invoke | KomuraSoft Blog

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

What is Media Foundation? Is it different from COM?

Media Foundation is a media processing platform for handling video and audio on Windows, and the API as a whole is not simply pure COM. That said, the boundaries between its components - source, transform, sink, activation, attributes, and callback - are expressed as COM interfaces, so IUnknown, HRESULT, GUIDs, and apartments naturally come up while you use it. The accurate way to see it is that Media Foundation is a media processing platform with COM deeply embedded in its boundary surfaces.

Why do I need both MFStartup and CoInitializeEx?

Because they play different roles. CoInitializeEx initializes the COM library, and MFStartup initializes the Media Foundation platform. COM initialization alone is not enough; Media Foundation needs its own initialization too. On real projects, deciding up front which thread uses Media Foundation, whether that thread is STA or MTA, and who owns MFStartup / MFShutdown and CoInitializeEx / CoUninitialize makes callbacks and UI integration much easier later.

How do I choose between the Source Reader, Sink Writer, Media Session, and MFTs?

If you want to pull frames or samples out of a file or camera, the Source Reader is the entry point; if you want to write generated audio or video out to a file, the Sink Writer is. If you want the platform to handle playback, stop, seek, A/V synchronization, and quality control, center your design on the Media Session. You move to MFTs when you need to plug your own decoder or transform into the pipeline, but the COM style contracts come to the foreground there, so it is better to first see which of the other three you actually need.

What should I watch out for with Media Foundation asynchronous callbacks?

Media Foundation's asynchronous processing uses work queues, and those threads are MTA, so the implementation becomes simpler if the application side also works in the MTA. It is important to make your IMFSourceReaderCallback implementation thread-safe and to avoid touching UI thread STA objects directly from the callback. If you need to update the UI, send only the result back to the UI thread. Also note that the Source Reader's synchronous or asynchronous mode is decided at creation time and cannot be switched afterwards.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.