KomuraSoft LLC
Contact
Menu

How to Burn Images and Text into MP4 Frames with Media Foundation

· Updated: · · Media Foundation, C++, Windows Development, GDI+, Direct2D, DirectWrite, H.264

Revision history (1 updates, last updated Sep 1, 2026)

A log of the changes made to this article. Where a pre-update version was archived, it stays readable at a permanent DOI link.

Retranslated as a full translation of the Japanese original. The previous English version was an abridgement that carried only part of the source, so sections, tables, Mermaid diagrams, figure captions and FAQ entries were missing. All of them have been restored to match the Japanese original, and the technical claims are the same as in the Japanese version. Read the version before this update (DOI: 10.5281/zenodo.21614523)
First published
Cite this article(DOI: 10.5281/zenodo.21614522)

This article is archived on Zenodo. Below are both the DOI that always resolves to the latest version and the DOI pinned to the version you are reading.

Go Komura (2026). How to Burn Images and Text into MP4 Frames with Media Foundation. KomuraSoft LLC. https://doi.org/10.5281/zenodo.21614522 https://comcomponent.com/en/blog/2026/03/16/009-media-foundation-overlay-image-text-on-mp4-frames/

DOI (latest version)
10.5281/zenodo.21614522
DOI (this version)
10.5281/zenodo.22217161

Logo watermarks, inspection results, equipment IDs, operator names, timestamps. The requirement to burn this kind of information into every frame of an MP4 video and produce a new MP4 is quite common in surveillance, inspection, audit trails, and analysis UIs.

But once you start working with Media Foundation, you are confronted with IMFSourceReader, IMFSample, IMFMediaBuffer, IMFTransform, and IMFSinkWriter, and it suddenly becomes hard to see where exactly you are supposed to overlay text or a PNG.

In this article, we first lay out the big picture - Source Reader -> drawing -> color conversion -> Sink Writer - and then provide a single-file sample you can paste straight into a Visual Studio C++ console application. The sample reads a given MP4, draws a given image plus the text HelloWorld onto every frame, and produces an output MP4.

Note that this sample prioritizes being something you can paste in and run immediately, so it uses a configuration that re-encodes the video only. You could cram audio remuxing into the same program, but since the topic of this article is burning an image and text into every frame, we focus on that first.

How this sample narrows its scopeA diagram showing that this sample prioritizes being pasted in and run as is, so it re-encodes video only and leaves audio remuxing as an extension to add after the main topic of burning in the overlay works.Prioritize paste and runRe-encode video onlyFocus on burning into every frameDefer audio remux to a later extension

Figure 1: Keep the first version minimal, so that only the core topic, burning in the overlay, has to work.

The code in this article is published on GitHub as a complete sample set (a single-file .cpp plus a CMake build configuration).

media-foundation-overlay-image-text-on-mp4-frames - komurasoft-blog-samples (GitHub)

Who This Is For and What You Need

This article is written for intermediate developers about to write Windows video processing in C++. It assumes you have touched the basics of COM (ComPtr, HRESULT, reference counting) and that Media Foundation is still ahead of you.

Here is what you need to run the sample. The details and the conditions on the input data are collected in section 5.

Item Requirement
OS Windows 10 / 11
Development environment A Visual Studio 2022 C++ console application
Build configuration x64
Precompiled headers Configure this .cpp so that it does not use them
Input video An ordinary MP4. The width and height must be even (because NV12 is 4:2:0)
Output A video-only MP4. There is no audio track

Terms to Know Up Front

From the table in section 3 onward, some English terms appear without explanation. Here is one line for each of them first.

Term Meaning
remux Rebuilding only the wrapper (the container) while leaving the compressed data inside untouched. Because nothing is re-encoded, neither picture nor sound quality degrades, and the processing stays light
topology The graph Media Foundation uses to express which component data flows from and to. It corresponds to a block diagram connecting sources, transforms, and sinks
custom MFT A Media Foundation Transform you write yourself. Implementing IMFTransform lets you plug an effect into the Media Foundation pipeline as a component
stride The number of bytes one row of an image occupies in memory. It does not necessarily match width * 4; there can be padding at the end of each row

1. The Short Answer First

  • The basic pattern for putting an image or text into every frame of an MP4 is decode with the Source Reader -> composite onto uncompressed frames -> convert colors if needed -> re-encode with the Sink Writer.
  • Placing the image or text itself is not Media Foundation’s job. It is more natural to think about this part with drawing APIs such as GDI+, Direct2D, DirectWrite, and WIC.
  • If you are writing back to MP4(H.264), you will often need a conversion stage that bridges RGB32 / ARGB32, which is easy to draw on, and NV12 / I420 / YUY2, which encoders accept readily.
  • If you want to get your first version working, the configuration Source Reader -> RGB32 -> draw with GDI+ -> NV12 -> Sink Writer is easy to follow.
  • If you want to prioritize speed and extensibility, moving toward D3D11 / DXGI surface -> Direct2D / DirectWrite -> Video Processor MFT -> Sink Writer gives you more headroom.

Knowledge map for this article

This article lays out a structure for burning images and text into every frame of an MP4 with Media Foundation and producing a new MP4. The pipeline decodes RGB32 frames with IMFSourceReader, composites the image and the text with GDI+ in the self-contained single-file sample, converts BGRA to NV12 because an H.264 encoder usually expects YUV input such as NV12, and then has IMFSinkWriter write the result into an MP4 as H.264. Differences in stride and in vertical orientation are absorbed with IMF2DBuffer::Lock2D and normalized to top-down BGRA, and ReadSample has to be checked on three points, the HRESULT, the flags, and the sample. For production the article presents extensions in stages, moving color conversion to the Video Processor MFT, replacing the drawing with Direct2D/DirectWrite, and factoring the work out into a custom MFT when reusability is needed, and it positions audio remux as a step to add only after the video burn-in has been stabilized first.

Burning images and text into MP4 frames with Media FoundationDiagram showing the relationships between RGB32 decoding with IMFSourceReader and compositing with GDI+, the conversion from RGB formats to NV12 that an H.264 encoder needs as input, MP4 output through IMFSinkWriter, absorbing stride and top-down versus bottom-up differences, extensions toward Direct2D/DirectWrite, the Video Processor MFT and a custom MFT, and the approach of leaving audio remux to a later stage.usesusesrecommended forrequiresusesrequiresusesincompatible withrequiresrequiresusesrecommended forrequiresusesrequiresusesmay causeshould come beforerecommended forrequiresBurning Overlays into Video FramesIMFSinkWriter (Sink Writer)IMFSourceReader (Source Reader)GDI+Direct2D / DirectWriteStride (Image Row Pitch)IMF2DBuffer::Lock2DTop-Down / Bottom-Up Image OrientationMFVideoFormat_RGB32H.264 Video Encoder (Media Foundation)NV12 Pixel FormatBGRA to NV12 Color ConversionVideo Processor MFTIMFSourceReader::ReadSampleNull Sample from ReadSampleAudio RemuxCustom MFTMedia Foundation

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (20 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

2. Why This Problem Is a Bit Tricky

“Putting text into a video” is actually four different topics mixed together.

  1. Containers vs. codecs An mp4 is a container, not the frames themselves. The contents are usually compressed data such as H.264 or H.265.

  2. Decoding / encoding While the data is still compressed, you cannot simply overlay text or a PNG with an ordinary 2D drawing API. You first need to get back to uncompressed frames.

  3. Drawing Text, logos, alpha-blended PNGs, and anti-aliased text rendering are not the responsibility of Media Foundation itself. This is the job of GDI+ or Direct2D / DirectWrite / WIC.

  4. Color spaces and pixel formats The format that is easy to draw on and the format the encoder prefers are not the same. This is where people quietly get stuck.

Putting it bluntly in one line: rather than “putting text in with Media Foundation,” the clearest mental model is “use Media Foundation to move frames around, use a drawing API to overlay things, then apply any needed color conversion before encoding.”

The four topics mixed togetherA diagram showing that the requirement to put text into a video mixes four topics together: containers and codecs, decoding and encoding, drawing, and color spaces and pixel formats.Put text into a videoContainers and codecsDecoding and encodingDrawingColor spaces and pixel formats

Figure 2: When you get stuck, first work out which of these topics you are actually in.

3. The Overview Table to Look at First

Approach Configuration Best suited for Watch out for
Get it working correctly first Source Reader -> RGB32 -> composite -> NV12 -> Sink Writer Batch processing, internal tools, initial implementations CPU-side copies and conversions tend to add up
Increase speed D3D11 / DXGI surface -> Direct2D / DirectWrite -> Video Processor MFT -> Sink Writer Long videos, high resolutions, bulk processing More D3D11 and DXGI management
Make it a reusable component Implement as a custom MFT and insert it into a topology Effects shared across multiple apps, integration into an MF pipeline Implementation, registration, and debugging get harder

The sample in this article is limited to the top row, the “get it working correctly first” configuration.

3.1 Processing Overview

input.mp4IMFSourceReaderUncompressed frameRGB32Draw image + HelloWorld with GDI+BGRA -> NV12 conversionIMFSinkWriteroutput.mp4Audio samplesCopy as isor re-encode

Figure 3: Pull frames out with the Source Reader, draw with GDI+, convert to NV12, and write back with the Sink Writer.

The important point here is that the drawing itself is not Media Foundation’s job. Media Foundation is responsible for moving frames in and out; placing the image and text is delegated to a drawing API.

4. How to Split Up the Pipeline

4.1 Receive the Input with IMFSourceReader

If the input is a file path, use MFCreateSourceReaderFromURL; if it is video data in memory, a clear approach is to create an IMFByteStream and use MFCreateSourceReaderFromByteStream.

The first thing to decide here is whether to receive frames in a format that is easy to draw on, or in a format aimed at the encoder.

  • If you want a simple implementation, use RGB32 or ARGB32
  • If you want encoding efficiency, use a YUV format such as NV12

That said, compositing text and PNGs is overwhelmingly easier to reason about in RGB formats, so receiving frames as RGB32 / ARGB32 is the easy first move.

First move on which format to receiveA diagram showing that RGB32 or ARGB32 is the choice when you want a simple implementation and NV12 or another YUV format when you want encoding efficiency, but because compositing text and PNGs is easier to reason about in RGB, receiving RGB first is the easy move.A simple implementationEncoding efficiencyWhat do you prioritizeReceive RGB32 / ARGB32Receive NV12 or another YUVCompositing is easier to reason about in RGB

Figure 4: When in doubt, start by receiving RGB so that drawing stays easy.

If you enable MF_SOURCE_READER_ENABLE_VIDEO_PROCESSING, the Source Reader performs YUV -> RGB32 conversion and deinterlacing for you. This is convenient at the “I just want to pull out frames and work with them” stage, but it tends to get heavy with long or high-resolution videos, so if you need speed in production it is worth revisiting the configuration later.

The trade-off of ENABLE_VIDEO_PROCESSINGA diagram showing that enabling MF_SOURCE_READER_ENABLE_VIDEO_PROCESSING lets the Source Reader handle YUV to RGB32 conversion and deinterlacing, but that it tends to get heavy for long or high-resolution videos, so the configuration is worth revisiting when production needs speed.Enable the flagDelegate YUV to RGB32 conversionDelegate deinterlacing tooGets heavy for long or high-resolution video

Figure 5: The convenience flag that makes extraction easy comes at a cost in speed.

4.2 Think About Image and Text Compositing in Terms of GDI+ or Direct2D / DirectWrite

You take the buffer out of the IMFSample received from Media Foundation and place a logo image and text on top of it.

This sample prioritizes being easy to paste as a single file, so it uses GDI+ for drawing.

  • It can load images
  • It can draw text
  • It requires relatively little extra setup
  • It fits easily into a single console-app .cpp

On the other hand, for workloads that process long videos or lots of 4K content, D3D11 + Direct2D + DirectWrite has more headroom. A natural progression is to use GDI+ for the first implementation and move to Direct2D / DirectWrite when you need to optimize for speed.

4.3 You Cannot Necessarily Write RGB32 Straight to H.264

This is where people get stuck most often.

When writing back to MP4(H.264), Microsoft’s H.264 encoder usually expects YUV-family input such as I420 / IYUV / NV12 / YUY2 / YV12. In other words, compositing in the easy-to-draw RGB32 / ARGB32 and then handing the result straight to IMFSinkWriter is not guaranteed to just work.

So in practice you need one of two conversions.

  • Insert a Video Processor MFT to do RGB32 / ARGB32 -> NV12
  • Implement your own RGB -> NV12 conversion

This sample prioritizes being a single self-contained file, so it takes the latter route with a hand-rolled conversion. In production, inserting a Video Processor MFT, which can handle color-space conversion, resizing, and deinterlacing all together, is also a strong option.

Two routes from RGB to NV12A diagram showing that after compositing in the easy-to-draw RGB32 or ARGB32 you need either a Video Processor MFT or a hand-rolled RGB to NV12 conversion, and that this sample picks the hand-rolled route to stay in a single file.Finished compositing in RGBConvert with a Video Processor MFTConvert to NV12 by handThe sample prioritizes staying in one fileA strong option in production

Figure 6: Assume the conversion stage is mandatory, and only choose which side carries it.

4.4 Write the Output with IMFSinkWriter

For video output, IMFSinkWriter is the easiest to work with.

The idea is simple: you configure two things separately,

  • The output stream type … the format you want written to the file Example: MFVideoFormat_H264
  • The input stream type … the format the app hands to the Sink Writer Example: MFVideoFormat_NV12

So from the Sink Writer’s perspective,

  • the app side hands over uncompressed NV12 frames
  • the Sink Writer encodes them to H.264 and writes them into the MP4

is the relationship.

How the Sink Writer divides its input and output typesA diagram showing that you configure the Sink Writer with NV12 as the input stream type the app hands over and H.264 as the output stream type written to the file, and that the Sink Writer performs the encoding and writes the MP4.The app hands over NV12 framesSink WriterEncode to H.264Write into the MP4Configure the input and output types separately

Figure 7: Configuring what you hand over separately from what gets written is the Sink Writer way.

4.5 Treating Audio Separately at First Keeps Things Tidy

Very often you only want to put a logo or text into the video and do not want to touch the audio at all.

In practice, the configuration

  • video stream only: Source Reader -> composite -> Sink Writer
  • audio stream: remux it while still compressed

is easy to work with.

However, since this sample focuses on burning an image and text into the frames, the output is a video-only MP4. A version that preserves audio is easier to follow if you add it later as an extension.

Thinking about video and audio separatelyA diagram showing that in practice only the video stream goes from the Source Reader through compositing to the Sink Writer while the audio stream is remuxed while still compressed, and that this sample keeps its focus by producing video-only output.Video streamComposite and send to the Sink WriterAudio streamRemux while still compressedNot covered by the sample

Figure 8: If only the video needs to change, leave the audio alone and carry it over in the container.

5. Assumptions and Usage for This Sample

The assumptions for this code are as follows.

  • Windows 10 / 11
  • A Visual Studio 2022 C++ console application
  • x64 build
  • This .cpp file does not use precompiled headers
  • The input video’s width and height are even
  • The input is an ordinary MP4 video file
  • The output is a video-only MP4
  • The image is in a format GDI+ can read, such as PNG / JPEG / BMP / GIF

NV12 is 4:2:0, so the width and height must be even. For that reason, this sample explicitly raises an error when those conditions are not met.

The assumption that width and height are evenA diagram showing that NV12 uses 4:2:0 subsampling so the input video width and height have to be even, and that this sample stops with an explicit error when that condition is not met.EvenAn odd value is presentNV12 is 4:2:0Width and height must be evenIs the input evenContinue processingStop with an explicit error

Figure 9: Better to reject input that breaks the assumptions at the door than to fail silently later.

5.1 Usage

  1. Create a Console App in Visual Studio
  2. Paste this .cpp in wholesale
  3. Set that .cpp file’s precompiled header option to “Not Using”
  4. Build for x64
  5. Run it as follows
OverlayMp4.exe input.mp4 overlay.png output.mp4
  • input.mp4 The source video
  • overlay.png The image to overlay
  • output.mp4 The output destination

The text string is hard-coded to HelloWorld in kOverlayText at the top of the code. Position and size can also be changed by adjusting the constants in the code. The modification that lets you pass the text as a command-line argument is in section 9.5.

5.2 Checking That It Actually Worked

“It finished without errors” and “it burned in the overlay correctly” are two different things. Looking at the following four things in order will catch most failures.

  1. Look at the frame count printed at the end. When this sample finishes, it prints Done. frames= followed by the number of frames it wrote. If that is far off from the total frame count of the input video, the ReadSample loop is dropping frames somewhere
  2. Compare the output file’s basic information against the input. Right-click the output MP4 in Explorer and open Properties > Details to see the length, frame width, frame height, and frame rate. If the length does not match the input, suspect the timestamp handling (7.4)
  3. Look at three places with your own eyes: the start, the middle, and the end. Checking only the first frame and calling it done will let you miss a bug where the overlay disappears partway through. The reliable approach is to extract still images at the same timestamp from both the input and the output and put them side by side; that procedure is described in “Extracting a Still Image from an MP4 at a Specific Time with Media Foundation”
  4. Look for color anomalies. If skin tones or the sky look unnatural, the coefficient choice in BgraToNv12 (BT.601 versus BT.709) may not match the input

For your first attempt, use a short MP4 of a few seconds and a PNG with crisp edges. Trying to get your first run working on a long video makes isolating problems take much longer.

Steps for confirming that it workedA diagram showing that finishing without errors is not the same as burning in the overlay correctly, so you check four things in order: the frame count, the basic information of the output file, a visual check at the start middle and end, and any color anomaly.Compare the frame count with the inputCompare length and size in the propertiesInspect the start, middle, and end by eyeCheck for unnatural colorsIf anything is off, suspect the coefficient choice

Figure 10: Finishing without errors and being correct are different things, so check four angles in order.

6. Single-File Code You Can Paste Straight into a .cpp

6.1 A Map of the Code

Here is the map first. The code is long, but the parts you actually need to read are just three functions - CopySampleToTopDownBgra, DrawOverlay, and BgraToNv12 - plus the loop in wmain. Everything else is initialization and cleanup.

Function / class Role More detail
ScopedMf / ScopedGdiplus Pair up the startup and shutdown of MFStartup and GDI+ using RAII -
ConfigureSourceReader Set the Source Reader output to RGB32 and pull out the width, height, fps, and default frame duration 4.1
GetDefaultStride Work out the default stride from the media type 7.2
BufferLock Lock the buffer through IMF2DBuffer when one is available, and through IMFMediaBuffer otherwise 7.2
CopySampleToTopDownBgra Absorb stride and vertical orientation, normalizing into top-down BGRA 7.2
DrawOverlay Draw the image and text with GDI+. This is the one and only drawing stage 4.2 / 7.1
BgraToNv12 Convert the finished BGRA into NV12 4.3 / 7.1
CreateNv12Sample Wrap the NV12 buffer in an IMFSample and attach a timestamp and duration 7.4
ChooseBitrate Decide the output bitrate from the input information -
CreateSinkWriter Configure the H.264 type on the output side and the NV12 type we hand over 4.4
The while loop in wmain Run one frame at a time while checking ReadSample’s HRESULT / flags / sample 7.3 / 7.4

Lining this up with the processing overview in section 3, CopySampleToTopDownBgra corresponds to the uncompressed frame, DrawOverlay to drawing with GDI+, and BgraToNv12 to the BGRA to NV12 conversion.

The three functions at the coreA diagram showing that of all this long code the parts you actually need to read are CopySampleToTopDownBgra, DrawOverlay, and BgraToNv12 plus the wmain loop, which correspond to normalizing the uncompressed frame, drawing, and converting to NV12.The wmain loop runs one frame at a timeCopySampleToTopDownBgraDrawOverlayBgraToNv12Normalize the uncompressed frameDraw the image and textPrepare it for encoding

Figure 11: Everything else is setup and teardown; the core is these three functions plus the loop.

6.2 The Full Code

#define NOMINMAX
#include <windows.h>
#include <mfapi.h>
#include <mfidl.h>
#include <mfreadwrite.h>
#include <mferror.h>
#include <gdiplus.h>
#include <wrl/client.h>

#include <algorithm>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <cwchar>
#include <iostream>
#include <stdexcept>
#include <string>
#include <vector>

#pragma comment(lib, "mfplat.lib")
#pragma comment(lib, "mfreadwrite.lib")
#pragma comment(lib, "mfuuid.lib")
#pragma comment(lib, "mf.lib")
#pragma comment(lib, "gdiplus.lib")

using Microsoft::WRL::ComPtr;

namespace
{
    const wchar_t* kOverlayText = L"HelloWorld";
    const float kMarginRatio = 0.03f;
    const float kImageMaxWidthRatio = 0.20f;
    const float kImageMaxHeightRatio = 0.20f;
    const float kMinFontPx = 24.0f;

    std::string HrToHex(HRESULT hr)
    {
        char buf[32]{};
        std::snprintf(buf, sizeof(buf), "0x%08X", static_cast<unsigned int>(hr));
        return std::string(buf);
    }

    void ThrowIfFailed(HRESULT hr, const char* message)
    {
        if (FAILED(hr))
        {
            throw std::runtime_error(std::string(message) + " failed. HRESULT=" + HrToHex(hr));
        }
    }

    void ThrowIfGdiplusError(Gdiplus::Status status, const char* message)
    {
        if (status != Gdiplus::Ok)
        {
            char buf[128]{};
            std::snprintf(buf, sizeof(buf), "%s failed. GDI+ status=%d", message, static_cast<int>(status));
            throw std::runtime_error(buf);
        }
    }

    BYTE ClampToByte(int value)
    {
        if (value < 0) return 0;
        if (value > 255) return 255;
        return static_cast<BYTE>(value);
    }

    class ScopedGdiplus
    {
    public:
        ScopedGdiplus()
        {
            Gdiplus::GdiplusStartupInput input;
            ThrowIfGdiplusError(Gdiplus::GdiplusStartup(&token_, &input, nullptr), "GdiplusStartup");
        }

        ~ScopedGdiplus()
        {
            if (token_ != 0)
            {
                Gdiplus::GdiplusShutdown(token_);
            }
        }

    private:
        ULONG_PTR token_ = 0;
    };

    class ScopedMf
    {
    public:
        ScopedMf()
        {
            ThrowIfFailed(CoInitializeEx(nullptr, COINIT_MULTITHREADED), "CoInitializeEx");
            comInitialized_ = true;

            ThrowIfFailed(MFStartup(MF_VERSION), "MFStartup");
            mfStarted_ = true;
        }

        ~ScopedMf()
        {
            if (mfStarted_)
            {
                MFShutdown();
            }

            if (comInitialized_)
            {
                CoUninitialize();
            }
        }

    private:
        bool comInitialized_ = false;
        bool mfStarted_ = false;
    };

    class BufferLock
    {
    public:
        explicit BufferLock(IMFMediaBuffer* buffer)
            : buffer_(buffer)
        {
            if (!buffer_)
            {
                throw std::runtime_error("BufferLock received a null buffer.");
            }

            buffer_.As(&buffer2D_);
        }

        HRESULT LockBuffer(LONG defaultStride, DWORD heightInPixels, BYTE** scanline0, LONG* actualStride)
        {
            if (scanline0 == nullptr || actualStride == nullptr)
            {
                return E_POINTER;
            }

            HRESULT hr = S_OK;

            if (buffer2D_)
            {
                hr = buffer2D_->Lock2D(scanline0, actualStride);
            }
            else
            {
                BYTE* data = nullptr;
                hr = buffer_->Lock(&data, nullptr, nullptr);
                if (SUCCEEDED(hr))
                {
                    *actualStride = defaultStride;
                    if (defaultStride < 0)
                    {
                        *scanline0 = data + (static_cast<LONG>(heightInPixels) - 1) * std::abs(defaultStride);
                    }
                    else
                    {
                        *scanline0 = data;
                    }
                }
            }

            locked_ = SUCCEEDED(hr);
            return hr;
        }

        ~BufferLock()
        {
            if (!locked_)
            {
                return;
            }

            if (buffer2D_)
            {
                buffer2D_->Unlock2D();
            }
            else
            {
                buffer_->Unlock();
            }
        }

    private:
        ComPtr<IMFMediaBuffer> buffer_;
        ComPtr<IMF2DBuffer> buffer2D_;
        bool locked_ = false;
    };

    struct VideoFormatInfo
    {
        UINT32 width = 0;
        UINT32 height = 0;
        UINT32 fpsNum = 0;
        UINT32 fpsDen = 0;
        UINT32 parNum = 1;
        UINT32 parDen = 1;
        LONG sourceStride = 0;
        LONGLONG defaultFrameDuration = 0;
        UINT32 bitrate = 0;
    };

    LONG GetDefaultStride(IMFMediaType* type)
    {
        LONG stride = 0;

        HRESULT hr = type->GetUINT32(MF_MT_DEFAULT_STRIDE, reinterpret_cast<UINT32*>(&stride));
        if (SUCCEEDED(hr))
        {
            return stride;
        }

        GUID subtype = GUID_NULL;
        UINT32 width = 0;
        UINT32 height = 0;

        ThrowIfFailed(type->GetGUID(MF_MT_SUBTYPE, &subtype), "GetGUID(MF_MT_SUBTYPE)");
        ThrowIfFailed(MFGetAttributeSize(type, MF_MT_FRAME_SIZE, &width, &height), "MFGetAttributeSize(MF_MT_FRAME_SIZE)");
        ThrowIfFailed(MFGetStrideForBitmapInfoHeader(subtype.Data1, width, &stride), "MFGetStrideForBitmapInfoHeader");
        ThrowIfFailed(type->SetUINT32(MF_MT_DEFAULT_STRIDE, static_cast<UINT32>(stride)), "SetUINT32(MF_MT_DEFAULT_STRIDE)");

        return stride;
    }

    UINT32 ChooseBitrate(IMFMediaType* nativeType, UINT32 width, UINT32 height, UINT32 fpsNum, UINT32 fpsDen)
    {
        UINT32 srcBitrate = 0;
        if (SUCCEEDED(nativeType->GetUINT32(MF_MT_AVG_BITRATE, &srcBitrate)) && srcBitrate > 0)
        {
            return srcBitrate;
        }

        const double fps = static_cast<double>(fpsNum) / static_cast<double>(fpsDen);
        double estimated = static_cast<double>(width) * static_cast<double>(height) * fps * 0.07;

        if (estimated < 1500000.0)
        {
            estimated = 1500000.0;
        }

        if (estimated > 25000000.0)
        {
            estimated = 25000000.0;
        }

        return static_cast<UINT32>(estimated);
    }

    VideoFormatInfo ConfigureSourceReader(IMFSourceReader* reader)
    {
        ThrowIfFailed(reader->SetStreamSelection(MF_SOURCE_READER_ALL_STREAMS, FALSE), "SetStreamSelection(all,false)");
        ThrowIfFailed(reader->SetStreamSelection(MF_SOURCE_READER_FIRST_VIDEO_STREAM, TRUE), "SetStreamSelection(video,true)");

        ComPtr<IMFMediaType> nativeType;
        ThrowIfFailed(reader->GetNativeMediaType(MF_SOURCE_READER_FIRST_VIDEO_STREAM, 0, &nativeType), "GetNativeMediaType(video)");

        ComPtr<IMFMediaType> requestedType;
        ThrowIfFailed(MFCreateMediaType(&requestedType), "MFCreateMediaType(video requested)");
        ThrowIfFailed(requestedType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video), "SetGUID(video requested major)");
        ThrowIfFailed(requestedType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_RGB32), "SetGUID(video requested subtype RGB32)");
        ThrowIfFailed(reader->SetCurrentMediaType(MF_SOURCE_READER_FIRST_VIDEO_STREAM, nullptr, requestedType.Get()), "SetCurrentMediaType(video RGB32)");

        ComPtr<IMFMediaType> currentType;
        ThrowIfFailed(reader->GetCurrentMediaType(MF_SOURCE_READER_FIRST_VIDEO_STREAM, &currentType), "GetCurrentMediaType(video)");

        VideoFormatInfo info;
        ThrowIfFailed(MFGetAttributeSize(currentType.Get(), MF_MT_FRAME_SIZE, &info.width, &info.height), "Get video frame size");

        HRESULT hr = MFGetAttributeRatio(currentType.Get(), MF_MT_FRAME_RATE, &info.fpsNum, &info.fpsDen);
        if (FAILED(hr))
        {
            ThrowIfFailed(MFGetAttributeRatio(nativeType.Get(), MF_MT_FRAME_RATE, &info.fpsNum, &info.fpsDen), "Get video frame rate");
        }

        if (info.fpsNum == 0 || info.fpsDen == 0)
        {
            throw std::runtime_error("Video frame rate is zero.");
        }

        hr = MFGetAttributeRatio(currentType.Get(), MF_MT_PIXEL_ASPECT_RATIO, &info.parNum, &info.parDen);
        if (FAILED(hr) || info.parNum == 0 || info.parDen == 0)
        {
            info.parNum = 1;
            info.parDen = 1;
        }

        info.sourceStride = GetDefaultStride(currentType.Get());
        info.defaultFrameDuration = (10000000LL * info.fpsDen) / info.fpsNum;
        if (info.defaultFrameDuration <= 0)
        {
            throw std::runtime_error("Calculated frame duration is invalid.");
        }

        info.bitrate = ChooseBitrate(nativeType.Get(), info.width, info.height, info.fpsNum, info.fpsDen);
        return info;
    }

    ComPtr<IMFSinkWriter> CreateSinkWriter(const std::wstring& outputPath, const VideoFormatInfo& videoInfo, DWORD* streamIndex)
    {
        if (streamIndex == nullptr)
        {
            throw std::runtime_error("streamIndex is null.");
        }

        ComPtr<IMFAttributes> attributes;
        ThrowIfFailed(MFCreateAttributes(&attributes, 1), "MFCreateAttributes(sink)");
        ThrowIfFailed(attributes->SetUINT32(MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, TRUE), "SetUINT32(MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS)");

        ComPtr<IMFSinkWriter> writer;
        ThrowIfFailed(MFCreateSinkWriterFromURL(outputPath.c_str(), nullptr, attributes.Get(), &writer), "MFCreateSinkWriterFromURL");

        ComPtr<IMFMediaType> outputType;
        ThrowIfFailed(MFCreateMediaType(&outputType), "MFCreateMediaType(video output)");
        ThrowIfFailed(outputType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video), "SetGUID(output major)");
        ThrowIfFailed(outputType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_H264), "SetGUID(output subtype H264)");
        ThrowIfFailed(outputType->SetUINT32(MF_MT_AVG_BITRATE, videoInfo.bitrate), "SetUINT32(output bitrate)");
        ThrowIfFailed(outputType->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive), "SetUINT32(output interlace)");
        ThrowIfFailed(MFSetAttributeSize(outputType.Get(), MF_MT_FRAME_SIZE, videoInfo.width, videoInfo.height), "MFSetAttributeSize(output frame size)");
        ThrowIfFailed(MFSetAttributeRatio(outputType.Get(), MF_MT_FRAME_RATE, videoInfo.fpsNum, videoInfo.fpsDen), "MFSetAttributeRatio(output fps)");
        ThrowIfFailed(MFSetAttributeRatio(outputType.Get(), MF_MT_PIXEL_ASPECT_RATIO, videoInfo.parNum, videoInfo.parDen), "MFSetAttributeRatio(output PAR)");
        ThrowIfFailed(writer->AddStream(outputType.Get(), streamIndex), "AddStream(video)");

        ComPtr<IMFMediaType> inputType;
        ThrowIfFailed(MFCreateMediaType(&inputType), "MFCreateMediaType(video input)");
        ThrowIfFailed(inputType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video), "SetGUID(input major)");
        ThrowIfFailed(inputType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_NV12), "SetGUID(input subtype NV12)");
        ThrowIfFailed(inputType->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive), "SetUINT32(input interlace)");
        ThrowIfFailed(MFSetAttributeSize(inputType.Get(), MF_MT_FRAME_SIZE, videoInfo.width, videoInfo.height), "MFSetAttributeSize(input frame size)");
        ThrowIfFailed(MFSetAttributeRatio(inputType.Get(), MF_MT_FRAME_RATE, videoInfo.fpsNum, videoInfo.fpsDen), "MFSetAttributeRatio(input fps)");
        ThrowIfFailed(MFSetAttributeRatio(inputType.Get(), MF_MT_PIXEL_ASPECT_RATIO, videoInfo.parNum, videoInfo.parDen), "MFSetAttributeRatio(input PAR)");
        ThrowIfFailed(writer->SetInputMediaType(*streamIndex, inputType.Get(), nullptr), "SetInputMediaType(video)");

        ThrowIfFailed(writer->BeginWriting(), "BeginWriting");
        return writer;
    }

    void CopySampleToTopDownBgra(IMFSample* sample, const VideoFormatInfo& videoInfo, std::vector<BYTE>& bgra)
    {
        ComPtr<IMFMediaBuffer> buffer;
        ThrowIfFailed(sample->ConvertToContiguousBuffer(&buffer), "ConvertToContiguousBuffer");

        BufferLock lock(buffer.Get());

        BYTE* scanline0 = nullptr;
        LONG actualStride = 0;
        ThrowIfFailed(lock.LockBuffer(videoInfo.sourceStride, videoInfo.height, &scanline0, &actualStride), "LockBuffer");

        const size_t dstStride = static_cast<size_t>(videoInfo.width) * 4;
        bgra.resize(dstStride * videoInfo.height);

        for (UINT32 y = 0; y < videoInfo.height; ++y)
        {
            const BYTE* srcRow = scanline0 + static_cast<LONG>(y) * actualStride;
            BYTE* dstRow = bgra.data() + static_cast<size_t>(y) * dstStride;
            std::memcpy(dstRow, srcRow, dstStride);

            for (UINT32 x = 0; x < videoInfo.width; ++x)
            {
                dstRow[static_cast<size_t>(x) * 4 + 3] = 0xFF;
            }
        }
    }

    void DrawOverlay(std::vector<BYTE>& bgra, UINT32 width, UINT32 height, Gdiplus::Image& overlayImage)
    {
        const INT stride = static_cast<INT>(width * 4);

        Gdiplus::Bitmap frameBitmap(
            static_cast<INT>(width),
            static_cast<INT>(height),
            stride,
            PixelFormat32bppPARGB,
            bgra.data());
        ThrowIfGdiplusError(frameBitmap.GetLastStatus(), "Create frame bitmap");

        Gdiplus::Graphics graphics(&frameBitmap);
        ThrowIfGdiplusError(graphics.GetLastStatus(), "Create graphics");

        graphics.SetCompositingMode(Gdiplus::CompositingModeSourceOver);
        graphics.SetCompositingQuality(Gdiplus::CompositingQualityHighQuality);
        graphics.SetInterpolationMode(Gdiplus::InterpolationModeHighQualityBicubic);
        graphics.SetSmoothingMode(Gdiplus::SmoothingModeAntiAlias);
        graphics.SetTextRenderingHint(Gdiplus::TextRenderingHintAntiAliasGridFit);

        const Gdiplus::REAL margin = std::max<Gdiplus::REAL>(16.0f, static_cast<Gdiplus::REAL>(height) * kMarginRatio);
        const Gdiplus::REAL maxImageW = static_cast<Gdiplus::REAL>(width) * kImageMaxWidthRatio;
        const Gdiplus::REAL maxImageH = static_cast<Gdiplus::REAL>(height) * kImageMaxHeightRatio;

        const Gdiplus::REAL srcW = static_cast<Gdiplus::REAL>(overlayImage.GetWidth());
        const Gdiplus::REAL srcH = static_cast<Gdiplus::REAL>(overlayImage.GetHeight());
        if (srcW <= 0.0f || srcH <= 0.0f)
        {
            throw std::runtime_error("Overlay image has invalid size.");
        }

        const Gdiplus::REAL imageScale =
            std::min<Gdiplus::REAL>(1.0f, std::min(maxImageW / srcW, maxImageH / srcH));

        const Gdiplus::REAL drawW = srcW * imageScale;
        const Gdiplus::REAL drawH = srcH * imageScale;

        Gdiplus::RectF imageRect(margin, margin, drawW, drawH);
        Gdiplus::SolidBrush imagePlate(Gdiplus::Color(96, 0, 0, 0));
        graphics.FillRectangle(
            &imagePlate,
            imageRect.X - 8.0f,
            imageRect.Y - 8.0f,
            imageRect.Width + 16.0f,
            imageRect.Height + 16.0f);

        graphics.DrawImage(&overlayImage, imageRect);

        const Gdiplus::REAL fontPx =
            std::max<Gdiplus::REAL>(kMinFontPx, static_cast<Gdiplus::REAL>(height) * 0.06f);

        Gdiplus::Font font(L"Segoe UI", fontPx, Gdiplus::FontStyleBold, Gdiplus::UnitPixel);
        ThrowIfGdiplusError(font.GetLastStatus(), "Create font");

        Gdiplus::StringFormat stringFormat;
        stringFormat.SetAlignment(Gdiplus::StringAlignmentNear);
        stringFormat.SetLineAlignment(Gdiplus::StringAlignmentNear);

        Gdiplus::RectF measureLayout(
            margin,
            static_cast<Gdiplus::REAL>(height) - margin - fontPx * 2.0f,
            static_cast<Gdiplus::REAL>(width) - margin * 2.0f,
            fontPx * 2.0f);

        Gdiplus::RectF measured;
        graphics.MeasureString(kOverlayText, -1, &font, measureLayout, &stringFormat, &measured);

        Gdiplus::RectF textBg(
            measured.X - 12.0f,
            measured.Y - 8.0f,
            measured.Width + 24.0f,
            measured.Height + 16.0f);

        Gdiplus::SolidBrush textPlate(Gdiplus::Color(128, 0, 0, 0));
        graphics.FillRectangle(&textPlate, textBg);

        Gdiplus::SolidBrush shadowBrush(Gdiplus::Color(220, 0, 0, 0));
        Gdiplus::RectF shadowLayout = measureLayout;
        shadowLayout.X += 2.0f;
        shadowLayout.Y += 2.0f;
        graphics.DrawString(kOverlayText, -1, &font, shadowLayout, &stringFormat, &shadowBrush);

        Gdiplus::SolidBrush textBrush(Gdiplus::Color(235, 255, 255, 255));
        graphics.DrawString(kOverlayText, -1, &font, measureLayout, &stringFormat, &textBrush);
    }

    void BgraToNv12(const BYTE* bgra, UINT32 width, UINT32 height, BYTE* nv12)
    {
        const bool useBt709 = (width > 1024 || height > 576);

        const int yR = useBt709 ? 47 : 66;
        const int yG = useBt709 ? 157 : 129;
        const int yB = useBt709 ? 16 : 25;

        const int uR = useBt709 ? -26 : -38;
        const int uG = useBt709 ? -87 : -74;
        const int uB = 112;

        const int vR = 112;
        const int vG = useBt709 ? -102 : -94;
        const int vB = useBt709 ? -10 : -18;

        BYTE* yPlane = nv12;
        BYTE* uvPlane = nv12 + static_cast<size_t>(width) * height;

        const size_t srcStride = static_cast<size_t>(width) * 4;

        for (UINT32 y = 0; y < height; ++y)
        {
            const BYTE* srcRow = bgra + static_cast<size_t>(y) * srcStride;
            BYTE* dstY = yPlane + static_cast<size_t>(y) * width;

            for (UINT32 x = 0; x < width; ++x)
            {
                const BYTE b = srcRow[x * 4 + 0];
                const BYTE g = srcRow[x * 4 + 1];
                const BYTE r = srcRow[x * 4 + 2];

                const int Y = ((yR * r + yG * g + yB * b + 128) >> 8) + 16;
                dstY[x] = ClampToByte(Y);
            }
        }

        for (UINT32 y = 0; y < height; y += 2)
        {
            const BYTE* row0 = bgra + static_cast<size_t>(y) * srcStride;
            const BYTE* row1 = bgra + static_cast<size_t>(y + 1) * srcStride;
            BYTE* dstUV = uvPlane + static_cast<size_t>(y / 2) * width;

            for (UINT32 x = 0; x < width; x += 2)
            {
                int b = 0;
                int g = 0;
                int r = 0;

                for (UINT32 dy = 0; dy < 2; ++dy)
                {
                    const BYTE* row = (dy == 0) ? row0 : row1;
                    for (UINT32 dx = 0; dx < 2; ++dx)
                    {
                        const UINT32 ix = x + dx;
                        b += row[ix * 4 + 0];
                        g += row[ix * 4 + 1];
                        r += row[ix * 4 + 2];
                    }
                }

                b = (b + 2) / 4;
                g = (g + 2) / 4;
                r = (r + 2) / 4;

                const int U = ((uR * r + uG * g + uB * b + 128) >> 8) + 128;
                const int V = ((vR * r + vG * g + vB * b + 128) >> 8) + 128;

                dstUV[x + 0] = ClampToByte(U);
                dstUV[x + 1] = ClampToByte(V);
            }
        }
    }

    ComPtr<IMFSample> CreateNv12Sample(
        const std::vector<BYTE>& bgra,
        const VideoFormatInfo& videoInfo,
        LONGLONG sampleTime,
        LONGLONG sampleDuration)
    {
        const DWORD bufferSize =
            static_cast<DWORD>(videoInfo.width * videoInfo.height * 3 / 2);

        ComPtr<IMFMediaBuffer> buffer;
        ThrowIfFailed(MFCreateMemoryBuffer(bufferSize, &buffer), "MFCreateMemoryBuffer");

        BYTE* dst = nullptr;
        DWORD maxLength = 0;
        DWORD currentLength = 0;
        ThrowIfFailed(buffer->Lock(&dst, &maxLength, &currentLength), "Lock(NV12 buffer)");

        try
        {
            BgraToNv12(bgra.data(), videoInfo.width, videoInfo.height, dst);
        }
        catch (...)
        {
            buffer->Unlock();
            throw;
        }

        ThrowIfFailed(buffer->Unlock(), "Unlock(NV12 buffer)");
        ThrowIfFailed(buffer->SetCurrentLength(bufferSize), "SetCurrentLength(NV12 buffer)");

        ComPtr<IMFSample> sample;
        ThrowIfFailed(MFCreateSample(&sample), "MFCreateSample");
        ThrowIfFailed(sample->AddBuffer(buffer.Get()), "AddBuffer(output sample)");
        ThrowIfFailed(sample->SetSampleTime(sampleTime), "SetSampleTime");
        ThrowIfFailed(sample->SetSampleDuration(sampleDuration), "SetSampleDuration");

        return sample;
    }
}

int wmain(int argc, wchar_t* argv[])
{
    if (argc != 4)
    {
        std::wcerr << L"Usage: OverlayMp4.exe <input.mp4> <overlayImage.png> <output.mp4>" << std::endl;
        return 1;
    }

    const std::wstring inputPath = argv[1];
    const std::wstring imagePath = argv[2];
    const std::wstring outputPath = argv[3];

    try
    {
        if (_wcsicmp(inputPath.c_str(), outputPath.c_str()) == 0)
        {
            throw std::runtime_error("Input and output paths must be different.");
        }

        ScopedMf mf;
        ScopedGdiplus gdiplus;

        ComPtr<IMFAttributes> readerAttributes;
        ThrowIfFailed(MFCreateAttributes(&readerAttributes, 1), "MFCreateAttributes(reader)");
        ThrowIfFailed(
            readerAttributes->SetUINT32(MF_SOURCE_READER_ENABLE_VIDEO_PROCESSING, TRUE),
            "SetUINT32(MF_SOURCE_READER_ENABLE_VIDEO_PROCESSING)");

        ComPtr<IMFSourceReader> reader;
        ThrowIfFailed(
            MFCreateSourceReaderFromURL(inputPath.c_str(), readerAttributes.Get(), &reader),
            "MFCreateSourceReaderFromURL");

        VideoFormatInfo videoInfo = ConfigureSourceReader(reader.Get());

        if ((videoInfo.width % 2) != 0 || (videoInfo.height % 2) != 0)
        {
            throw std::runtime_error(
                "This sample requires even video width and height because NV12 is 4:2:0.");
        }

        Gdiplus::Image overlayImage(imagePath.c_str());
        ThrowIfGdiplusError(overlayImage.GetLastStatus(), "Load overlay image");

        DWORD videoStreamIndex = 0;
        ComPtr<IMFSinkWriter> writer =
            CreateSinkWriter(outputPath, videoInfo, &videoStreamIndex);

        std::vector<BYTE> bgra;
        LONGLONG firstTimestamp = -1;
        unsigned long long frameCount = 0;

        while (true)
        {
            DWORD flags = 0;
            LONGLONG timestamp = 0;
            ComPtr<IMFSample> inputSample;

            ThrowIfFailed(
                reader->ReadSample(
                    MF_SOURCE_READER_FIRST_VIDEO_STREAM,
                    0,
                    nullptr,
                    &flags,
                    &timestamp,
                    &inputSample),
                "ReadSample(video)");

            if ((flags & MF_SOURCE_READERF_CURRENTMEDIATYPECHANGED) != 0)
            {
                throw std::runtime_error("Dynamic video format change is not supported in this sample.");
            }

            if ((flags & MF_SOURCE_READERF_NATIVEMEDIATYPECHANGED) != 0)
            {
                throw std::runtime_error("Native video format change is not supported in this sample.");
            }

            if ((flags & MF_SOURCE_READERF_STREAMTICK) != 0)
            {
                if (firstTimestamp < 0)
                {
                    firstTimestamp = timestamp;
                }

                ThrowIfFailed(
                    writer->SendStreamTick(videoStreamIndex, timestamp - firstTimestamp),
                    "SendStreamTick");
            }

            if (inputSample)
            {
                if (firstTimestamp < 0)
                {
                    firstTimestamp = timestamp;
                }

                LONGLONG duration = 0;
                if (FAILED(inputSample->GetSampleDuration(&duration)) || duration <= 0)
                {
                    duration = videoInfo.defaultFrameDuration;
                }

                CopySampleToTopDownBgra(inputSample.Get(), videoInfo, bgra);
                DrawOverlay(bgra, videoInfo.width, videoInfo.height, overlayImage);

                ComPtr<IMFSample> outputSample =
                    CreateNv12Sample(bgra, videoInfo, timestamp - firstTimestamp, duration);

                ThrowIfFailed(
                    writer->WriteSample(videoStreamIndex, outputSample.Get()),
                    "WriteSample(video)");

                ++frameCount;
            }

            if ((flags & MF_SOURCE_READERF_ENDOFSTREAM) != 0)
            {
                break;
            }
        }

        ThrowIfFailed(writer->Finalize(), "Finalize");

        std::wcout
            << L"Done. frames=" << frameCount
            << L", output=" << outputPath
            << std::endl;

        return 0;
    }
    catch (const std::exception& ex)
    {
        std::cerr << ex.what() << std::endl;
        return 1;
    }
}

7. Points to Keep in Mind When Reading This Implementation

7.1 The Format That Is Easy to Draw on and the Format the Encoder Accepts Are Different

This sample uses the following flow.

  • Source Reader output: RGB32
  • Drawing: GDI+
  • Sink Writer input: NV12

The reason is simple: RGB formats are easy to work with when overlaying text and PNGs, and NV12 is easy to hand off to H.264 encoding.

When reading the implementation, it becomes easier to follow if you split it into a “drawing stage” and a “prepare-for-encoding stage.”

7.2 Stride and Vertical Orientation Are Normalized Before Drawing

Video frames are not necessarily laid out in memory the way they appear on screen.

  • The stride may not match width * 4
  • The image may be stored upside down
  • IMF2DBuffer and IMFMediaBuffer are handled slightly differently

For that reason, this code first normalizes into a top-down BGRA buffer before drawing. Getting this sorted out up front lets the drawing code stay quite straightforward.

Why normalize before drawingA diagram showing that variations such as a stride that does not match four times the width, a flipped vertical orientation, and different handling between IMF2DBuffer and IMFMediaBuffer are absorbed up front by normalizing into a top-down BGRA buffer before drawing.The stride does not matchNormalize into top-down BGRAThe orientation can be flippedBuffer types are handled differentlyThe drawing code stays straightforward

Figure 12: Absorb the memory-layout variation in one place and the drawing side never has to know about it.

7.3 With ReadSample, Check the Flags and the sample, Not Just the HRESULT

ReadSample can return S_OK with sample == nullptr. Typical cases are

  • MF_SOURCE_READERF_STREAMTICK
  • MF_SOURCE_READERF_ENDOFSTREAM
  • other stream events

So the loop needs to look at all three together: the HRESULT, the flags, and the inputSample. In particular, if you miss STREAMTICK or ENDOFSTREAM, downstream timeline handling tends to break.

The three things to check with ReadSampleA diagram showing that ReadSample can return S_OK with a null sample, typically on stream events such as STREAMTICK or ENDOFSTREAM, so the loop checks the HRESULT, the flags, and the sample together.ReadSample returnsCheck the HRESULTCheck the flagsCheck whether a sample is presentS_OK can still come with nullptr

Figure 13: Do not judge from the return value alone; handle each frame with all three signals.

7.4 It Is Safer to Carry Over Timestamps and Durations from the Input

Timestamps are in 100-ns units. Also, the duration has to be retrieved separately from the IMFSample.

Rather than assuming a fixed fps and adding a hard-coded increment each time, it is more robust to carry over the input sample’s timestamp / duration as much as possible. This sample does exactly that, falling back to a default value computed from the fps only when the duration cannot be obtained.

Handling timestamp and durationA diagram showing that instead of adding a hard-coded increment on the assumption of a fixed frame rate you carry over the input sample timestamp and duration as much as possible and fall back to a default computed from fps only when the duration cannot be obtained.AvailableNot availableRead them from the input sampleWas a duration availableCarry it over as isUse the fps-derived defaultMore robust than a hard-coded increment

Figure 14: Do not invent times yourself; carrying them over from the input is the default stance.

7.5 GDI+ Is Lightweight to Adopt, but There Is a Next Step for Long or High-Resolution Content

GDI+ is very well suited to a single-file sample, but for workloads processing long videos or lots of 4K content, D3D11 + Direct2D + DirectWrite can be the better choice.

  • First get the whole pipeline working with GDI+
  • Then, if needed, replace it with Direct2D / DirectWrite
  • Move color conversion to a Video Processor MFT or the GPU side

A staged progression like this lets you extend the system without breaking the design.

A staged path for the drawing APIA diagram showing the staged path of getting the whole pipeline working with GDI+ first, replacing it with Direct2D and DirectWrite when that becomes necessary, and moving color conversion to a Video Processor MFT or the GPU without breaking the design.Get the whole pipeline working with GDI+Replace with Direct2D if neededMove color conversion to an MFT or the GPULong or bulk 4K work is the next stage

Figure 15: Start with what is cheapest to adopt, then replace the parts where performance actually matters.

7.6 This Sample Is Limited to Video Only

If you also pile audio into the same article, the focus gets diluted. For that reason, this sample concentrates on burning an image and text into the video frames, and the output is a video-only MP4.

In practice, the next step is to grow it into

  • video only: Source Reader -> composite -> Sink Writer
  • audio: remux it while still compressed

which is an easy configuration to manage.

8. If the “Given Video Data” Is an In-Memory MP4 Byte Sequence Rather Than a File

The code in this article uses MFCreateSourceReaderFromURL, so the input is a file path.

But if the requirement is “do the same thing to mp4 bytes received from an API,” the thinking does not change. Only the entry point changes.

  • Prepare an IStream or a custom stream
  • Hand it to the Source Reader as an IMFByteStream
  • From there on it is the same: RGB32 -> draw -> NV12 -> Sink Writer

In other words, the essence is not how the video data is held, but how you draw onto each decoded frame.

Only the entry point changes for byte-sequence inputA diagram showing that a file path goes through MFCreateSourceReaderFromURL while an in-memory MP4 byte sequence is wrapped in an IMFByteStream and handed to the Source Reader, after which the flow from RGB32 through drawing and NV12 to the Sink Writer is the same.File pathSource ReaderIn-memory byte sequenceWrap it in an IMFByteStreamThe rest of the flow is the same

Figure 16: However the data is held, only the entry stage changes.

9. Growing It for Production

9.1 Add Audio Remuxing

The most practical first extension is to preserve the audio as is. Re-encode only the video and write the audio back in the same format while still compressed; this meets the requirement without adding much implementation.

The Sink Writer explicitly supports the combination of taking compressed input and writing it to the output in the same format, as a remux path with no re-encoding. There are three places to add.

  1. Receive the audio stream while it is still compressed. Enable it with reader->SetStreamSelection(MF_SOURCE_READER_FIRST_AUDIO_STREAM, TRUE), then pass the type you obtained from GetNativeMediaType straight into SetCurrentMediaType. Specifying the native type is how you tell the Source Reader that you do not want it decoded
  2. Set that same type on the Sink Writer as both the input and the output. Pass the same media type to writer->AddStream(audioType.Get(), &audioStreamIndex) and writer->SetInputMediaType(audioStreamIndex, audioType.Get(), nullptr)
  3. Use the same timestamp baseline as the video. Subtract the same firstTimestamp that the video path in section 6.2 uses from the audio samples as well. Using separate baselines here makes the audio drift out of sync with the picture

The ReadSample call also changes from its current form, which names the video stream only, to one that uses MF_SOURCE_READER_ANY_STREAM and dispatches on the stream index it returns.

Note that the Sink Writer does not resample audio or resize and frame-rate-convert video unless the encoder provides it. If the MP4 sink cannot accept the input audio format, you need re-encoding rather than a remux.

The three places audio remuxing addsA diagram showing that preserving the audio means adding three things: receiving the audio stream while it is still compressed, setting the same media type as both input and output on the Sink Writer, and using the same timestamp baseline as the video.Receive the audio while still compressedSet the same type as input and outputShare the timestamp baseline with the videoSeparate baselines make the audio drift

Figure 17: Remuxing adds only three places, but do not forget to line up the timestamp baseline.

9.2 Insert a Video Processor MFT

This sample converts BGRA -> NV12 by hand to stay self-contained in a single file, but in production, inserting a Video Processor MFT is also a very strong option.

With the Video Processor MFT, it becomes easier to handle

  • color-space conversion
  • resizing
  • deinterlacing
  • frame-rate conversion

all in one place.

9.3 Replace GDI+ with Direct2D / DirectWrite

For overlays such as logo images, subtitles, and timestamps, GDI+ is often sufficient, but if you need to squeeze out performance, Direct2D / DirectWrite has the edge.

In particular, if you have conditions such as

  • high resolution
  • long durations
  • large numbers of videos
  • a future move toward a GPU path

then a configuration based on D3D11 / DXGI surface comes into view.

9.4 Consider a Custom MFT Once It Becomes a “Video Effect You Want to Reuse”

In Media Foundation, effects can be implemented as an IMFTransform. So if you want to reuse the same overlay processing across multiple apps or pipelines, a custom MFT is a clean choice.

However, as a first implementation,

  • you must satisfy the IMFTransform contract
  • input/output media-type management increases
  • registration and debugging get harder

so in practice it is usually easier to first get things working correctly with Source Reader + compositing + Sink Writer, and extract an MFT when you actually need one.

When to consider a custom MFTA diagram showing the decision flow of first getting things working correctly with the Source Reader, compositing, and the Sink Writer, and extracting a custom MFT only once you want to reuse the same overlay processing across multiple apps or pipelines.Want it in several appsOne app is enoughGet the current design working correctly firstDo you now want to reuse itExtract it as a custom MFTKeep the current designImplementation, registration, and debugging get harder

Figure 18: Componentizing is clean, but waiting until you need it is not too late.

9.5 Turn the String into an Argument So You Can Draw Japanese

The sample’s text is fixed to HelloWorld in kOverlayText. If you are burning in equipment IDs or operator names, this is the first thing you will want to turn into an argument. Drawing Japanese also means changing the font.

There are four places to change.

1. Add the font names you want to use to the anonymous namespace. Keep the existing kOverlayText as the default value.

    const wchar_t* kOverlayText = L"HelloWorld";          // Existing. Used as the default when the argument is omitted
    const wchar_t* kFontFamilyName = L"Yu Gothic UI";     // A face that renders Japanese
    const wchar_t* kFallbackFontFamilyName = L"Segoe UI"; // For machines that do not have the one above

2. Let DrawOverlay take the string to draw.

    void DrawOverlay(
        std::vector<BYTE>& bgra,
        UINT32 width,
        UINT32 height,
        Gdiplus::Image& overlayImage,
        const std::wstring& overlayText)      // Added

3. Inside DrawOverlay, swap out the font creation and the string references. Replace the original Gdiplus::Font font(L"Segoe UI", ...) line with the following.

        // If the requested face is not installed, fall back to the default face
        Gdiplus::FontFamily preferred(kFontFamilyName);
        Gdiplus::FontFamily fallback(kFallbackFontFamilyName);
        const Gdiplus::FontFamily& family = preferred.IsAvailable() ? preferred : fallback;
        if (!family.IsAvailable())
        {
            throw std::runtime_error("Neither the preferred nor the fallback font family is installed.");
        }

        Gdiplus::Font font(&family, fontPx, Gdiplus::FontStyleBold, Gdiplus::UnitPixel);
        ThrowIfGdiplusError(font.GetLastStatus(), "Create font");

On top of that, replace every kOverlayText passed to MeasureString and to the two DrawString calls with overlayText.c_str(). If you do not fix all three, the shadow will still show the old text.

4. Receive the argument in wmain and pass it to DrawOverlay.

    if (argc < 4 || argc > 5)
    {
        std::wcerr
            << L"Usage: OverlayMp4.exe <input.mp4> <overlayImage.png> <output.mp4> [text]"
            << std::endl;
        return 1;
    }

    const std::wstring inputPath = argv[1];
    const std::wstring imagePath = argv[2];
    const std::wstring outputPath = argv[3];
    const std::wstring overlayText = (argc == 5) ? std::wstring(argv[4]) : std::wstring(kOverlayText);

Then change the call inside the loop to this.

                DrawOverlay(bgra, videoInfo.width, videoInfo.height, overlayImage, overlayText);

There are two things to watch out for when handling Japanese.

  • If you write Japanese as a literal in the .cpp, save the source as UTF-8 with BOM or build with /utf-8 on MSVC. Get this wrong and the text comes out garbled. If you pass the text as a command-line argument instead, wmain receives it as UTF-16, so the problem does not arise
  • A font face is not guaranteed to be installed. Always provide a fallback, as in the code above. If a machine without the font silently switches to a different face, it becomes hard to track down why the layout shifted
The changes needed to draw JapaneseA diagram showing that turning the string into an argument so Japanese can be drawn changes four places, namely adding font name constants, adding a parameter to DrawOverlay, switching to font creation with a fallback, and receiving the argument in wmain, with garbled text and missing fonts as the pitfalls.Add the font name constantsAdd a parameter to DrawOverlayCreate the font with a fallbackReceive the argument in wmain and pass it onDrop to the default when the face is missing

Figure 19: Four places change, and only the font setup and the character encoding are likely to trip you up.

10. Summary

When burning images or text into every frame of an MP4 with Media Foundation, breaking the problem into these four parts gives you a clear view.

  • Extract: IMFSourceReader
  • Draw: GDI+ or Direct2D / DirectWrite
  • Convert into a format the encoder accepts: NV12, etc.
  • Write back: IMFSinkWriter

And if what you want is “a sample you can paste entirely into one .cpp and run as is,” then a configuration like the one in this article,

Source Reader -> RGB32 -> image + HelloWorld with GDI+ -> BGRA to NV12 -> Sink Writer

is quite natural.

If you grow it for production next, thinking in this order keeps things from falling apart.

  1. Add audio remuxing
  2. Replace GDI+ with Direct2D / DirectWrite
  3. Move the NV12 conversion to a Video Processor MFT or the GPU side
  4. Move to a D3D11 surface-based design for long, high-resolution content
  5. Extract a custom MFT if you need reusability

If you try to do everything at once, COM, strides, color spaces, and surface management all hit you at the same time. Getting it working stage by stage first, then strengthening only the parts you need later, makes both the design and the debugging considerably easier.

The order for growing it in productionA diagram showing that adding audio remuxing, replacing GDI+ with Direct2D and DirectWrite, moving the NV12 conversion to a Video Processor MFT or the GPU, moving to a D3D11 surface based design for long or high-resolution content, and extracting a custom MFT when reuse is needed is an order that holds together.Add audio remuxingReplace the drawing with Direct2DMove the conversion to an MFT or the GPUMove to a D3D11 surface based designExtract a custom MFT if needed

Figure 20: Avoid doing everything at once and strengthen one stage at a time in this order.

12. References

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

What is the basic flow for burning an image or text into every frame of an MP4 with Media Foundation?
The basic pattern is: decode with the Source Reader, composite onto uncompressed frames, convert the color format if needed, and re-encode with the Sink Writer. Placing the image or text is not Media Foundation's job at all; that belongs to drawing APIs such as GDI+, Direct2D/DirectWrite, and WIC. For a first working version, the configuration Source Reader to RGB32, draw with GDI+, convert to NV12, then Sink Writer is the easiest to follow, and if you want to prioritize speed and extensibility, moving toward D3D11/DXGI surfaces with Direct2D/DirectWrite gives you more headroom.
Can I hand RGB32 frames straight to H.264 encoding?
Not necessarily. Microsoft's H.264 encoder usually expects YUV-family input such as I420, IYUV, NV12, YUY2, or YV12, so after compositing in the easy-to-draw RGB32/ARGB32 you will usually need a conversion stage. Either insert a Video Processor MFT to do RGB32 to NV12, or implement the RGB to NV12 conversion yourself. NV12 is also 4:2:0, so the frame width and height have to be even.
Should I use GDI+ or Direct2D for drawing the overlay?
For a first implementation, GDI+ suits a single-file sample: it loads images, draws text, and needs little extra setup. For long videos, 4K, or bulk processing, D3D11 plus Direct2D and DirectWrite can have the performance edge. Getting the whole pipeline working with GDI+ first, then replacing it with Direct2D/DirectWrite when you start tuning for speed and moving color conversion to a Video Processor MFT or the GPU, is a staged progression that extends the system without breaking the design.
What do I need to watch out for when using IMFSourceReader::ReadSample?
ReadSample can return S_OK and still leave sample as nullptr. Typical cases are stream events such as MF_SOURCE_READERF_STREAMTICK and MF_SOURCE_READERF_ENDOFSTREAM. The loop therefore has to check three things together: the HRESULT, the flags, and the sample. Timestamps are in 100-ns units, and rather than adding a hard-coded increment on the assumption of a fixed frame rate, carrying over the input sample's timestamp and duration wherever possible is more robust.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.

Contact
Back to the Blog