Vector graphics rendering in Unreal Engine: Part 1

Integrating a comprehensive 2D graphics API into Unreal Engine for custom UI rendering.

The first phase of getting our custom Unreal Engine UI framework off the ground is integrating a solid 2D graphics API and using it as the basis for some core widget types. As a quick recap: the primary goal here is to enable faster UI design iterations in Unreal by allowing developers to specify things like border styles and rounded corners directly in the UMG editor, instead of needing to bake out static raster images for 9-sliced texture sampling (or grow a big enough brain to intuit the math for an equivalent signed distance function — very cool technique, but not particularly easy to understand without a lot of practice.)

Selecting and integrating an API

We could build such an API ourselves, but we’re aiming to get something up and running in the span of a weekend or so, not several years of concerted effort, so we’ll pull something ready-made off the shelf. The most popular choice for this type of thing is probably Google’s Skia , which powers the rendering engines for Chrome, Android, Flutter, and plenty of non-Google projects as well.

Skia does carry some notable drawbacks, though. It’s not trivial to compile, and it’s not really clear how we could convince its GPU backend to play nicely with Unreal’s hardware abstractions, even if they use the same platform-specific API under the hood (which is not a given). The path of least resistance will be to just use Skia’s CPU-driven raster backend to write to a buffer in host memory and upload it to an Unreal render target on the GPU. This is far from ideal, but it should serve our needs adequately as a proof-of-concept.

In the longer term, it’s probably worth investigating Vello as an alternative. It’s explicitly designed to be API-agnostic, so we could theoretically translate the Recordings it generates to Unreal RHI commands relatively easily — but using it at all would require some Rust/C++ FFI bindings and translating a bunch of WGSL shaders to Unreal’s not-quite-but-almost HLSL. I don’t have enough experience in either of those areas to be comfortable going down that road as a first attempt.

Compiling Skia

Being a large C++ library with a bespoke build system, this is equal parts fiddly and finicky. I’m not going to walk through the process step-by-step here, because Skia’s own documentation is a better resource for that , but it did take some considerable Googling to figure out how to correctly build it as a dynamic library on Windows with Clang, so I’ll provide the build script I eventually arrived at in case it proves useful. This works on my particular dev environment, but no guarantees it’ll work for you — make sure you have the correct dependencies installed and that the paths point to valid locations if you run into trouble.

build.bat
set PATH=D:\depot_tools;%PATH%

@REM NOTE: The line breaks below are just for readability -- you'll need to
@REM       replace them with spaces before running. You'll also want to set
@REM       `is_debug=false` and build a second time (to a separate folder!)
@REM       for release.

.\bin\gn gen out/x64/Debug/Shared --args="
	is_debug=true
	is_component_build=true
	is_official_build=false
	target_cpu=\"x64\"
	win_vc=\"C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\"
	clang_win=\"C:\\Program Files\\LLVM\"
	extra_cflags=[\"/MD\"]
	extra_cflags_cc=[\"/MD\"]"

ninja -C out/x64/Debug/Shared

Setting up a 3rd-party Unreal plugin

A plugin that hosts a 3rd-party library can be bootstraped from the Unreal Editor by navigating to Edit > Plugins, pressing the + Add button at the top left, and selecting the Third Party Library template at the bottom of the list. (Alternatively, if you’re using Rider you can do the same thing from the IDE by right-clicking on your project and selecting Add > Unreal Plugin…)

At this point, it would only be responsible to note that I have a really tenuous grasp on what I’m doing here, so proceed with caution, and if any of this is terribly foolish, please let me know (and if you run into problems at this stage, Godspeed to you — let me know if you figure it out).

These are the steps that I took:

  • Copy the DLL from <skia>/out/x64/Release/Shared/skia.dll to <plugin>/Binaries/ThirdParty/Skia/Win64/
  • Copy the entire contents of <skia>/out/x64/Debug/Shared/ (except the gen folder) into <plugin>/Source/ThirdParty/Skia/Win64/Debug/
  • Copy the entire contents of <skia>/out/x64/Release/Shared/ (except the gen folder) into <plugin>/Source/ThirdParty/Skia/Win64/Release/
  • Create an includes folder under <plugin>/Source/ThirdParty/Skia/
    • Copy <skia>/include to <...>/Skia/includes/include
    • Copy <skia>/client_utils to <...>/Skia/includes/client_utils
    • Copy <skia>/modules to <...>/Skia/includes/modules
    • Copy <skia>/third_party to <...>/Skia/includes/third_party
    • Copy <skia>/out/x64/Release/Shared/gen/skia.h into <...>/Skia/includes/

Configure the Unreal build script:

Skia.build.cs
using System.IO;
using UnrealBuildTool;

public class Skia : ModuleRules
{
	public Skia(ReadOnlyTargetRules target)
		: base(target)
	{
		Type = ModuleType.External;
		
		// NOTE: If you want to support other platforms, you will need to compile
		//       Skia for them and use the approriate platform-specific API macro
		//       and paths here
		if (Target.Platform == UnrealTargetPlatform.Win64)
		{
			// Define the SK_API macro
			PublicDefinitions.Add("SK_API=__declspec(dllimport)");
			
			// Add the import library
			string configFolder = Target.Configuration switch {
				UnrealTargetConfiguration.Debug
					or UnrealTargetConfiguration.DebugGame
					or UnrealTargetConfiguration.Development => "Debug",
				_ => "Release",
			};
			PublicAdditionalLibraries.Add(Path.Combine(
				ModuleDirectory,
				"Win64", configFolder, "skia.dll.lib"
			));
			
			// Delay-load the DLL, so we can load it in the right place first
			PublicDelayLoadDLLs.Add("skia.dll");
			
			// Ensure that the DLL is staged along with the executable
			RuntimeDependencies.Add("$(PluginDir)/Binaries/ThirdParty/Skia/Win64/skia.dll");
		}
		
		PublicIncludePaths.AddRange(new [] {
			Path.Combine(ModuleDirectory, "includes"),
		});
	}
}

Then load the DLL in the plugin’s main module:

Public/<MainModule>.h
#pragma once

#include "CoreMinimal.h"
#include "Modules/ModuleManager.h"


class FMainModuleModule : public IModuleInterface
{
public:
	void StartupModule() override;
	void ShutdownModule() override;
	
private:
	void* m_SkiaHandle = nullptr;
};
Private/<MainModule>.cpp
#include "<MainModule>.h"

#include "include/core/SkColor.h"
#include "Interfaces/IPluginManager.h"


void FMainModuleModule::StartupModule()
{
	// Get the base directory of this plugin
	FString baseDir = IPluginManager::Get().FindPlugin("<Plugin>")->GetBaseDir();
	
	// Add on the relative location of the third-party DLL and load it
	FString libPath;
#if PLATFORM_WINDOWS
	libPath = FPaths::Combine(*baseDir, TEXT("Binaries/ThirdParty/Skia/Win64/skia.dll"));
#endif
	
	if (!libPath.IsEmpty())
		m_SkiaHandle = FPlatformProcess::GetDllHandle(*libPath);
	
	if (m_SkiaHandle != nullptr)
	{
		// Simple sanity check to make sure the DLL loaded correctly
		SkScalar hsb[3] { 200.f, 1.f, 1.f };
		SkColor argb = SkHSVToColor(hsb);
		
		if (argb == 0xFF00AAFF)
		{
			UE_LOG(LogTemp, Log, TEXT("Successfully loaded skia.dll"));
		}
		else
		{
			UE_LOG(LogTemp, Error, TEXT("Loading of skia.dll produced unexpected result"));
		}
	}
	else
	{
		UE_LOG(LogTemp, Error, TEXT("Failed to load skia.dll"));
	}
}

void FMainModuleModule::ShutdownModule()
{
	if (m_SkiaHandle != nullptr)
	{
		FPlatformProcess::FreeDllHandle(m_SkiaHandle);
		m_SkiaHandle = nullptr;
	}
}

IMPLEMENT_MODULE(FMainModuleModule, <MainModule>)

(Note: If it wasn’t obvious, everything wrapped in angle braces above — and the MainModule part of the class name — should be replaced with the actual plugin or module names.)

On startup, we load the Skia DLL and validate it by calling a simple function from core/SkColor.h. If you see Successfully loaded skia.dll in Unreal Engine’s Output Log after starting up the editor, give yourself a pat on the back! Now we can remove the test code from the startup method (everything after the GetDllHandle call) and start painting some pixels.

Creating the base widgets

Before we get started, add these dependencies to your main module’s build script so we don’t need to worry about playing wack-a-mole with linker errors when we try to build the project later:

<MainModule>.Build.cs
using UnrealBuildTool;

public class MainModule : ModuleRules
{
	public MainModule(ReadOnlyTargetRules target)
		: base(target)
	{
		PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;
		
		PublicDependencyModuleNames.AddRange(new [] {
			"Core",
			"CoreUObject",
			"Engine",
			"Projects",
			"RenderCore",
			"Slate",
			"SlateCore",
			"Skia",
			"UMG",
		});
		
		PrivateDependencyModuleNames.AddRange(new [] {
			"RHI",
		});
	}
}

Sketching out some architecture

The difficulty of maintaining two different classes for every widget type is one of the high-level pain points of Slate/UMG that I highlighted at the outset of this project. We won’t be substantially addressing that here, but we can take a small step in that direction by defining the right boundaries of responsibility for our canvas-drawing widgets.

In the classic Slate/UMG setup, the underlying Slate widget is responsible for virtually all of the heavy lifting. It handles layout by first computing its “desired size,” and then arranging the positions of its children in a second pass. Finally, it paints itself to the viewport by issuing a command to Slate’s rendering interface (which is then translated to commands for the underlying render hardware interface). The UMG widget just provides an Unreal Editor interop layer, allowing a designer to manipulate UPropertys which are passed through to the Slate widget.

We’re going to take a different approach. Our base Slate widget will allocate a render target and bind it to an FSlateBrush, which we’ll use for painting. It’ll create an SkSurface for that render target, which can be drawn to through the SkCanvas API. But instead of the Slate widget using the SkCanvas directly, it’ll hold an FOnDraw delegate with the following signature:

void OnDraw(SkCanvas& canvas, FInt32Vector2 size, FVector2f scale);

Our UWidget derivatives will bind a method to that delegate to issue the actual draw commands, which means we only need the one base Slate widget to cover all of the rendering that can be done through this type of 2D graphics API. In the long run, that’s virtually everything — but for now, since we’re constrained to Skia’s CPU raster backend and may eventually need to switch to a completely different API to take advantage of hardware acceleration, we’ll treat this as a proof-of-concept and keep the scope fairly limited.

Implementing the Slate base widget

Authoring Slate widgets can be a pretty fraught process due to sparse documentation. Taking some advice from Ben UI , I managed to piece together a working solution through trial and error, starting by reverse-engineering a built-in widget that does something similar to what we’re trying to accomplish. But it’s worth noting that this may not be the ideal way to solve this particular problem. (Also, a full-on guide to authoring Slate widgets is well beyond the scope of this post — for that, Ben’s site linked above is as good a resource as there is on the topic.)

Essentially, what we want to do is paint our widget by simply displaying a render target, and only update the pixels in that render target when its existing pixels are invalidated for some reason. This is basically the same thing that’s done by the built-in SRetainerWidget , which draws its children to a render target to prevent unnecessary re-paints. The behavior of that widget is far more complex than what we need here, but we’ll borrow some of its key ideas.

To start, we’ll sketch out the basic interface for our widget. It will only have two inputs for now: OnDraw, which binds the canvas-drawing delegate described above, and DesiredSizeOverride, to allow the user to explicitly set the widget’s dimensions.

Public/SCanvasWidget.h
#pragma once

#include "CoreMinimal.h"

class SkCanvas;


DECLARE_DELEGATE_ThreeParams(FOnDraw,
	SkCanvas&,
	FInt32Vector2/* size */,
	FVector2f/* scale */)

class API SCanvasWidget : public SLeafWidget
{
	SLATE_DECLARE_WIDGET(SCanvasWidget, SLeafWidget)
	
public:
	SLATE_BEGIN_ARGS(SCanvasWidget) {}
		SLATE_EVENT(FOnDraw, OnDraw)
		SLATE_ATTRIBUTE(TOptional<FVector2D>, DesiredSizeOverride)
	SLATE_END_ARGS()
	
	SCanvasWidget();
	
	void Construct(const FArguments& args);
	
	int32 OnPaint(
		const FPaintArgs& args,
		const FGeometry& geo,
		const FSlateRect& rect,
		FSlateWindowElementList& elements,
		int32 layer,
		const FWidgetStyle& style,
		bool parentEnabled)
		const override;
		
	void RequestRedraw();
	
	FOnDraw OnDraw;
	
protected:
	FVector2D ComputeDesiredSize(float) const override;
	
private:
	bool m_NeedsRedraw = false;
	FSlateBrush m_Brush;
	TSlateAttribute<TOptional<FVector2D>> m_DesiredSizeOverride;
};

And some standard Slate widget boilerplate:

Private/SCanvasWidget.cpp
#include "SCanvasWidget.h"


SLATE_IMPLEMENT_WIDGET(SCanvasWidget)

void SCanvasWidget::PrivateRegisterAttributes(FSlateAttributeInitializer& init)
{
	SLATE_ADD_MEMBER_ATTRIBUTE_DEFINITION_WITH_NAME(init,
		"DesiredSizeOverride",
		m_DesiredSizeOverride,
		EInvalidateWidgetReason::Layout);
}

SCanvasWidget::SCanvasWidget()
	// Rider will spuriously highlight this as an error. It's not. I promise.
	: m_DesiredSizeOverride(*this)
{}

void SCanvasWidget::Construct(const FArguments& args)
{
	OnDraw = args._OnDraw;
	m_DesiredSizeOverride.Assign(*this, args._DesiredSizeOverride);
}

RequestRedraw will simply allow our UWidget derivatives to signal that they need to be drawn again, e.g. because one of their user-configurable properties has been updated. We’ll use it to set the corresponding flag, which will be checked before executing the OnDraw delegate in OnPaint.

Private/SCanvasWidget.cpp
// ...

void SCanvasWidget::RequestRedraw()
{
	m_NeedsRedraw = true;
}

To compute our desired size, we’ll simply defer to the DesiredSizeOverride input if it’s been set. Otherwise, we’ll return 0x0 (meaning that we’ll derive our size from Slate’s layout algorithm).

Private/SCanvasWidget.cpp
// ...

FVector2D SCanvasWidget::ComputeDesiredSize(float) const
{
	if (const auto& sizeOverride = m_DesiredSizeOverride.Get())
		return *sizeOverride;
	
	return FVector2D::ZeroVector;
}

Handling the render target texture is where we’ll take some inspiration from the built-in SRetainerWidget. The class we’ll want to use for the render target, UTextureRenderTarget2D , derives from UObject, which means that its lifetime should be managed by Unreal’s garbage collector. SWidget-derived classes live entirely outside of the whole UObject ecosystem, so we can’t just throw a raw UTextureRenderTarget2D pointer onto our SCanvasWidget and expect everything to be fine. Instead, we’ll delegate the memory-management tasks expected by Unreal to a separate class which we can store safely on our SCanvasWidget.

Public/SCanvasWidget.h
// ...
class SkCanvas;


namespace CanvasWidget {

class FRenderResources
	: public FDeferredCleanupInterface
	, public FGCObject
{
public:
	FRenderResources() = default;
	
	void Initialize();
	
	UTextureRenderTarget2D* RenderTarget() const;
	
	FString GetReferencerName() const override;
	void AddReferencedObjects(FReferenceCollector& gc) override;
	
private:
	UTextureRenderTarget2D* m_RenderTarget = nullptr;
};

} // namespace CanvasWidget

// ...

class API SCanvasWidget : public SLeafWidget
{
	SLATE_DECLARE_WIDGET(SCanvasWidget, SLeafWidget)
	
public:
	// ...
	~SCanvasWidget();
	// ...
private:
	// ...
	CanvasWidget::FRenderResources* m_RenderResources = nullptr;
};
Private/SCanvasWidget.cpp
#include "SCanvasWidget.h"

#include "Engine/TextureRenderTarget2D.h"


namespace CanvasWidget {

void FRenderResources::Initialize()
{
	if (m_RenderTarget != nullptr)
		return;
	
	auto* rt = NewObject<UTextureRenderTarget2D>();
	rt->ClearColor = FLinearColor::Transparent;
	rt->RenderTargetFormat = RTF_RGBA8_SRGB; // Skia's expected format
	
	m_RenderTarget = rt;
}

UTextureRenderTarget2D* FRenderResources::RenderTarget() const
{
	return m_RenderTarget;
}

FString FRenderResources::GetReferencerName() const
{
	return "CanvasWidget::FRenderResources";
}

void FRenderResources::AddReferencedObjects(FReferenceCollector& gc)
{
	gc.AddReferencedObject(m_RenderTarget);
}

} // namespace CanvasWidget

// ...

SCanvasWidget::SCanvasWidget()
	: m_DesiredSizeOverride(*this)
	, m_RenderResources(new CanvasWidget::FRenderResources)
{}

SCanvasWidget::~SCanvasWidget()
{
	BeginCleanup(m_RenderResources);
}

void SCanvasWidget::Construct(const FArguments& args)
{
	// ...
	m_RenderResources->Initialize();
	m_Brush.SetResourceObject(m_RenderResources->RenderTarget());
}

// ...

We’ll also go ahead and setup the SkSurface that will be synced to our render target. UTextureRenderTarget2D is a GPU resource, and (due to the API limitations I mentioned earlier) we can’t tell Skia to write to it directly. So we’ll also need to keep a buffer in host memory for the SkSurface, which we’ll upload to the render target resource whenever OnDraw is called.

Public/SCanvasWidget.h
// ...
class SkSurface;


namespace CanvasWidget {

class FRenderResources // ...
{
public:
	// ...
	SkSurface& Surface() const;
	// ...
private:
	// ...
	TSharedPtr<SkSurface> m_Surface = nullptr;
	TArray<uint8> m_PxBuffer;
};

} // namespace CanvasWidget
// ...

We could return SkSurface as a raw pointer from our getter, but (for reasons that will be more clear later) if our SCanvasWidget tries to access it before it’s been created, it will mean something has gone unrecoverably wrong. So rather than null-checking the pointer in SCanvasWidget and silently refusing to render anything if that happens, we’ll just assert that the shared pointer is valid at access-time and return a reference to the wrapped instance.

Private/SCanvasWidget
// ...
SkSurface& FRenderResources::Surface() const
{
	check(m_Surface.IsValid());
	return *m_Surface;
}
// ...

Before we can create the SkSurface, we need to know what size it should be. ComputeDesiredSize tells Slate what size the widget would like to be, all else being equal — but the actual render size is ultimately determined by Slate’s layout algorithm, which takes many variables into account (like, for example, what type of layout slot the widget is being rendered in and how that slot’s layout properties are configured by the user). We won’t have that information until OnPaint is called by the Slate runtime, where we can derive it from the FGeometry argument it’s invoked with.

Private/SCanvasWidget.cpp
// ...
int32 SCanvasWidget::OnPaint(
	const FPaintArgs& args,
	const FGeometry& geo,
	const FSlateRect& rect,
	FSlateWindowElementList& elements,
	int32 layer,
	const FWidgetStyle& style,
	bool parentEnabled)
	const
{
	FPaintGeometry paintGeo = geo.ToPaintGeometry();
	FSlateRenderTransform transform = paintGeo.GetAccumulatedRenderTransform();
	
	FVector2f scale = transform.GetMatrix().GetScale().GetVector();
	FVector2f localSize (paintGeo.GetLocalSize());
	FIntPoint renderSize = (localSize * scale).IntPoint();
	
	uint32 rtWidth = FMath::Abs(renderSize.X);
	uint32 rtHeight = FMath::Abs(renderSize.Y);
	
	// Make sure we have a valid size for the render target before going any further
	
	if (FMath::Max(rtWidth, rtHeight) > GetMax2DTextureDimension())
	{
		UE_LOG(LogTemp, Error,
			TEXT("The requested size for SCanvasWidget is too large: %i x %i"),
			rtWidth, rtHeight);
		
		return layer;
	}
	
	if (rtWidth == 0 || rtHeight == 0)
	{
		UE_LOG(LogTemp, Warning,
			TEXT("The requested size for SCanvasWidget has a zero dimension: %i x %i"),
			rtWidth, rtHeight);
		
		return layer;
	}
	
	// TODO
}

Since FRenderResources is managing the render target, we need a way to pass it the computed dimensions. If the render dimensions change for whatever reason without our corresponding UWidget explicitly requesting a redraw (e.g. due to a layout or screen resolution change), we also need FRenderResources to signal that to the SCanvasWidget, so that it can force a redraw with the new dimensions.

We’ll add a SetDimensions method to FRenderResources, which will return a boolean to indicate that an update is needed. (I don’t love the fact that it’s super non-obvious what that return value actually means, but we’ll add a documentation comment to alleviate any potential confusion.)

Public/SCanvasWidget.h
// ...
class FRenderResources // ...
{
public:
	// ...
	/** @returns true if the surface needs to be re-drawn. */
	UE_NODISCARD bool SetDimensions(uint32 width, uint32 height, FVector2f scale);
	// ...
private:
	// ...
	uint32 m_Width = 0;
	uint32 m_Height = 0;
	FVector2f m_Scale { 1.f, 1.f };
};
// ...
Private/SCanvasWidget.cpp
// ...
bool FRenderResources::SetDimensions(uint32 width, uint32 height, FVector2f scale)
{
	bool needsRedraw = false;
	
	if (m_Width != width || m_Height != height)
	{
		needsRedraw = true;
		m_Width = width;
		m_Height = height;
		
		// (Re)size the render target
		if (m_RenderTarget->GameThread_GetRenderTargetResource()
			&& m_RenderTarget->OverrideFormat == PF_B8G8R8A8)
		{
			m_RenderTarget->ResizeTarget(width, height);
		}
		else
		{
			m_RenderTarget->InitCustomFormat(width, height, PF_B8G8R8A8, false);
			m_RenderTarget->UpdateResourceImmediate();
		}
		
		// (Re)set the pixel buffer and Skia surface
		if (m_Surface.IsValid())
			m_Surface.Reset();
		
		auto imageInfo = SkImageInfo::Make(
			width, height,
			kBGRA_8888_SkColorType,
			kUnpremul_SkAlphaType);
		
		size_t bufferSize = imageInfo.computeByteSize(imageInfo.minRowBytes());
		m_PxBuffer.SetNumZeroed(bufferSize);
		
		m_Surface = Skia::MakeShareable(
			SkSurface::MakeRasterDirect(
				imageInfo,
				m_PxBuffer.GetData(),
				imageInfo.minRowBytes()));
	}
	
	if (!FMath::IsNearlyEqual(m_Scale.X, scale.X)
		|| !FMath::IsNearlyEqual(m_Scale.Y, scale.Y))
	{
		needsRedraw = true;
		m_Scale = scale;
	}
	
	return needsRedraw;
}
// ...

One important side note here: Many of Skia’s Make static methods return an sk_sp<T>, which is Skia’s take on std::shared_ptr<T>. For whatever reason, I’ve found that Unreal classes do not like storing sk_sp<T>s as fields. This may have something to do with the timing of the runtime DLL load versus when the headers are parsed, but whatever the case, I found it more convenient to just make a utility function to swap the sk_sp<T>s with Unreal-flavored TSharedPtr<T>s instead of trying to debug the issue:

Public/SkiaUtil.h
#pragma once

#include "CoreMinimal.h"
#include "include/core/SkRefCnt.h"

namespace Skia {

template <typename T>
FORCEINLINE TSharedPtr<T> MakeShareable(sk_sp<T> ptr)
{
	T* raw = ptr.release();
	return ::MakeShareable<T>(raw);
}

} // namespace Skia

Now, we can continue with our OnPaint implementation by setting the render resources dimensions and handling invalidation.

Private/SCanvasWidget.cpp
// ...
int32 SCanvasWidget::OnPaint(/*...*/) const
{
	// ...
	
	if (m_RenderResources->SetDimensions(rtWidth, rtHeight, scale))
	{
		m_Brush.ImageSize = FVector2f(rtWidth, rtHeight);
		m_NeedsRedraw = true;
	}
	
	// TODO
}

We’ll also need to mark m_NeedsRedraw and m_Brush as mutable, since we’re updating them in OnPaint, which is a const method. I don’t love having to do this, but since those fields are private implementation details that have no real impact beyond our own internal logic, it seems relatively harmless.

Public/SCanvasWidget.h
// ...
class API SCanvasWidget : public SLeafWidget
{
	// ...
private:
	mutable bool m_NeedsRedraw = false;
	mutable FSlateBrush m_Brush;
	// ...
};

Before we can invoke the OnDraw delegate, we need to make sure that FRenderResources is ready to be drawn to. We’ll add a method to check for that:

Public/SCanvasWidget.h
// ...
class FRenderResources // ...
{
public:
	// ...
	bool IsDrawable() const;
	// ...
};
// ...
Private/SCanvasWidget.cpp
// ...
bool FRenderResources::IsDrawable() const
{
	return (m_RenderTarget != nullptr
		&& m_Surface.IsValid()
		&& !m_PxBuffer.IsEmpty()
		&& m_Width > 0
		&& m_Height > 0);
}
// ...

After OnDraw is executed, we’ll need to upload the SkSurface’s pixel buffer to our render target on the GPU. We’ll add an Update method to FRenderResources to handle that.

Public/SCanvasWidget.h
// ...
class FRenderResources // ...
{
public:
	// ...
	void Update();
	// ...
};
// ...
Private/SCanvasWidget.cpp
// ...
void FRenderResources::Update()
{
	m_Surface->flush();
	
	UTextureRenderTarget2D* rt = m_RenderTarget;
	FUpdateTextureRegion2D region { 0, 0, 0, 0, m_Width, m_Height };
	size_t srcPitch = m_Surface->imageInfo().minRowBytes();
	uint8* pxData = m_PxBuffer.GetData();
	
	ENQUEUE_RENDER_COMMAND(UpdateCanvasBuffer)(
		[region, rt, srcPitch, pxData](FRHICommandList&)
		{
			FRHITexture* tex2d = rt
				->GetRenderTargetResource()
				->GetTextureRHI()
				->GetTexture2D();
			
			RHIUpdateTexture2D(tex2d, 0, region, srcPitch, pxData);
		});
}
// ...

Finally, we can finish our OnPaint implementation:

Private/SCanvasWidget.cpp
int32 SCanvasWidget::OnPaint(/*...*/) const
{
	// ...
	
	if (m_NeedsRedraw
		&& m_RenderResources->IsDrawable()
		&& OnDraw.IsBound())
	{
		SkSurface& surf = m_RenderResources->Surface();
		SkCanvas& canvas = *surf.getCanvas();
		FInt32Vector2 size { surf.width(), surf.height() };
		
		OnDraw.Execute(canvas, size, scale);
		m_RenderResources->Update();
		m_NeedsRedraw = false;
	}
	
	FSlateDrawElement::MakeBox(elements, layer, paintGeo, &m_Brush);
	
	return layer;
}

Fixing a concurrency issue

If you tried to take our SCanvasWidget now and start rendering UI with it, you might find that it mostly works, but with some weirdly inconsistent errors popping up in the Output Log claiming a size mismatch between the render target and the data we’re trying to upload to it.

No, it’s not an issue with our logic — at least, not in the way you might be thinking. What we’re seeing is a concurrency bug. Because the lambda in FRenderResources::Update is executed on Unreal’s render thread, but everything else happens on the game thread, it’s possible that our code will try to resize the render target and pixel buffer while the UpdateCanvasBuffer render command is still in flight, causing it to be invalid by the time it gets executed.

We can fix it by adding an atomic boolean flag to FRenderResources to serve as a sort of ad-hoc mutex. When that flag is set, we just need to avoid modifying the render target, which only happens in one place.

Public/SCanvasWidget.h
// ...
class FRenderResources // ...
{
	// ...
private:
	// ...
	FThreadSafeBool m_Locked = false;
};
// ...
Private/SCanvasWidget.cpp
// ...
bool FRenderResources::SetDimensions(uint32 width, uint32 height, FVector2f scale)
{
	if (m_Locked)
	{
		// There are RHI commands for the render target already in-flight, so we
		// need to defer any updates until after they're complete.
		return false;
	}
	
	// ...
}

void FRenderResources::Update()
{
	m_Surface->flush();
	
	// Lock the resource so that it doesn't get resized from the game thread
	// while there are RHI commands in-flight.
	FThreadSafeBool& locked = m_Locked;
	locked.AtomicSet(true);
	
	// ...
	
	ENQUEUE_RENDER_COMMAND(UpdateCanvasBuffer)(
		[region, rt, srcPitch, pxData, &locked](FRHICommandList&)
		{
			{
				FRHITexture* tex2d = rt
					->GetRenderTargetResource()
					->GetTextureRHI()
					->GetTexture2D();
				
				RHIUpdateTexture2D(tex2d, 0, region, srcPitch, pxData);
			}
			
			locked.AtomicSet(false);
		});
}
// ...

UMG base widget and checking our work

With any luck, that Slate widget will last us a good long while before it needs to be revisited. Now we can go ahead and create the UMG widget that will serve as the base for our actual vector-drawing widgets. There’s not much to this — mostly, we’re just establishing the API for deriving classes to override.

Public/CanvasWidget.h
#pragma once

#include "CoreMinimal.h"
#include "Components/Widget.h"

#include "CanvasWidget.generated.h"

class SCanvasWidget;
class SkCanvas;


UCLASS()
class API UCanvasWidget : public UWidget
{
	GENERATED_BODY()
	using Self = UCanvasWidget;
	
public:
	void SynchronizeProperties() override;
	void ReleaseSlateResources(bool) override;
	
protected:
	TSharedRef<SWidget> RebuildWidget() override;
	
	/**
	 * @param size The surface size measured in hardware device pixels.
	 * @param scale The logical Slate UI scale. Includes the ratio of hardware 
	 *    pixels to Slate units, as well as any render transforms applied.
	 */
	virtual void OnDraw(SkCanvas& canvas, FInt32Vector2 size, FVector2f scale);
	
	TSharedPtr<SCanvasWidget> SlateWidget;
};
Private/CanvasWidget.cpp
#include "CanvasWidget.h"

#include "SCanvasWidget.h"
#include "include/core/SkCanvas.h"
#include "include/core/SkPaint.h"


void UCanvasWidget::SynchronizeProperties()
{
	Super::SynchronizeProperties();
	
	TSharedPtr<SWidget> safeWidget = GetCachedWidget();
	if (safeWidget.IsValid())
	{
		auto* widget = (SCanvasWidget*)safeWidget.Get();
		widget->RequestRedraw();
	}
	else if (SlateWidget.IsValid())
	{
		SlateWidget->RequestRedraw();
	}
}

void UCanvasWidget::ReleaseSlateResources(bool)
{
	SlateWidget.Reset();
}

TSharedRef<SWidget> UCanvasWidget::RebuildWidget()
{
	SlateWidget = SNew(SCanvasWidget)
		.OnDraw_UObject(this, &Self::OnDraw);
	
	return SlateWidget.ToSharedRef();
}

void UCanvasWidget::OnDraw(SkCanvas& canvas, FInt32Vector2 size, FVector2f scale)
{
	SkPaint paint;
	paint.setColor(0xFF'FF00FF);
	paint.setStyle(SkPaint::kFill_Style);
	
	canvas.clear(SK_ColorTRANSPARENT);
	canvas.drawRect(SkRect::MakeWH(size.X, size.Y), paint);
}

Try booting up the editor, creating a new widget blueprint, and adding a UCanvasWidget to the widget tree. (You might need to wrap it in a Size Box or some other layout container that will give it dimensions, since we zeroed out the desired size by default.) If everything’s working correctly, you should see a bright magenta box in the UMG designer. Now we’re cooking with gas!

Feel free to take some time to toy around with the Skia API to see what sort of stuff it’s capable of. As long as you don’t modify any header files, you should be able to hot-reload any C++ changes for quick feedback in the editor (though you may need to close and reopen the UMG designer).

When you’re ready to move on, we’ll want to mark the class as Abstract and remove the placeholder OnDraw implementation. But before doing that, make sure you don’t save (or delete) any Canvas Widgets from widget blueprints before closing the editor — I’m not sure what would happen if you left a Canvas Widget in the tree and then disallowed it from instantiating with the Abstract specifier, but probably nothing good!

Public/CanvasWidget.h
// ...
UCLASS(Abstract)
class API UCanvasWidget : public UWidget
{
	// ...
	virtual void OnDraw(SkCanvas& canvas, FInt32Vector2 size, FVector2f scale) {}
	// ...
};
Private/CanvasWidget.cpp
// ...
TSharedRef<SWidget> UCanvasWidget::RebuildWidget()
{
	SlateWidget = SNew(SCanvasWidget);
	return SlateWidget.ToSharedRef();
}

Up next, we’ll create a couple of actually practical UMG widgets using Skia as a rendering engine: one that implements a tiny part of the CSS spec (rectangles with configurable fill color, independent corner radii and borders), and another that can render simple SVG images.