Mastering RAW 9 in Core Image: A Comprehensive Tutorial

Apple’s RAW 9 engine introduces massive quality jumps for image processing across iOS, iPadOS, macOS, and visionOS. Built on a tiled Core ML model and powered by the Apple Neural Engine (ANE), RAW 9 combines demosaicing and denoising into a single, highly optimized on-device step.

This tutorial breaks down how the RAW pipeline works, how to implement RAW 9, how to tune performance for interactive editing or exporting, and how to leverage advanced CIImageProcessor APIs.

1. Understanding the RAW Pipeline

Unlike standard formats like JPEG or HEIF, a RAW image file contains the unprocessed, uncompressed data captured directly by a camera's digital sensor. To display a RAW file on a screen, Core Image passes it through a multi-stage imaging pipeline.

Figure 1: Digital Sensor Mosaic (Bayer Filter Pattern Grid)

The Core Image Processing Steps

1. Unpacking and Parsing: The system extracts the file's metadata and reads the raw sensor values. At this point, the image is stored in a mosaic pattern (typically a Bayer filter array), where each individual pixel location only holds a single color channel: Red, Green, or Blue.
2. Demosaicing: This step interpolates the missing color data. It analyzes neighboring pixels to calculate the missing two color channels for every single pixel location, resulting in full RGB data.
3. Denoising: Cleans up the image by filtering out three distinct types of digital artifacts:
   • Photon Noise: Shot noise caused by statistical variations in light arrival.
   • Read Noise: Electrical noise introduced during the sensor's readout phase.
   • Thermal Noise: Heat-generated interference caused by long exposures or warm sensors.
4. Convolutions (Sharpening): Mathematical filters pass over the image data to sharpen edges and boost local contrast.
5. Final Adjustments: The pipeline applies white balance, exposure corrections, and color transformations to output a visually pleasing image.

2. Implementing RAW 9 with CIRAWFilter

RAW 9 is not enabled by default to maintain backward compatibility. You must explicitly query and opt in using the CIRAWFilter API.

Checking Support and Opting In

import CoreImage

// 1. Initialize your RAW filter with an image URL
guard let rawFilter = CIRAWFilter(imageURL: rawImageURL) else { return }

// 2. Verify that the system supports RAW 9
if rawFilter.supportedDecoderVersions.contains(CIRAWFilterOption.decoderVersion9) {
    // 3. Explicitly opt into RAW 9
    rawFilter.setValue(CIRAWFilterOption.decoderVersion9, forKey: kCIInputDecoderVersionKey)
}

Checking Camera Model Compatibility

Not every camera model supports the Core ML-driven RAW 9 engine immediately. You can check compatibility programmatically:

// Returns an array of supported camera models for a specific decoder version
let supportedModels = CIRAWFilter.supportedCameraModels(forVersion: CIRAWFilterOption.decoderVersion9)

💡 Note: The supported camera list includes major professional vendors and expands over-the-air via OS updates. Cameras that capture native DNG (like iPhones shooting Apple ProRAW) support RAW 9 automatically.

Calibrated Property Changes in RAW 9

RAW 9 simplifies your editing UI code because its underlying Core ML model handles complex artifacts automatically.

Property	Status in RAW 9	Behavior / Action
Exposure	Fully Supported	Controls overall image brightness. Works better than in prior versions.
Luminance Noise Reduction	Fully Supported	Adjusts the fine-grained luma noise texture.
Sharpness / Contrast	Fully Supported	Controls edge definition and local contrast.
Color Noise Reduction	Deprecated	Ignored. The Core ML model cleans up chroma noise automatically.
Detail / Moire Noise Reduction	Unsupported	No longer needed or functional in RAW 9.

You can check if a control is available at runtime using:

if rawFilter.isSupportedProperty(kCIInputColorNoiseReductionAmountKey) {
    // Show UI control if true (will return false for RAW 9)
}

3. Performance Best Practices

Because RAW 9 evaluates a heavy machine learning model hundreds of times across an image, strategic resource management is critical. Optimization depends on your app's explicit use case.

Scenario A: Interactive Editing

Definition: A single RAW file is repeatedly updated and re-rendered at screen resolution while a user drags editing sliders.

• Use the Scale Factor: Don't render a 60-megapixel image on a 3-megapixel phone screen. Set scaleFactor on CIRAWFilter to downscale the render work to match the display size.
• Enable Intermediate Caching: Maintain a single CIContext for your viewing pipeline and set cacheIntermediates to true. This instructs Core Image to cache the heavy Core ML demosaic/denoise output. Sub-second tweaks to sliders like exposure or contrast will skip the Core ML model entirely and render instantly.
• Add the Extended Virtual Addressing Entitlement: Add this entitlement to your application's capability settings. It allows the system to allocate larger chunks of virtual memory, maximizing Core Image's caching potential.
• Render Directly to Metal: Stream your output straight into an MTKView. Metal allows concurrent frame processing, scheduling the next frame's GPU workloads before the current frame finishes presenting.

Scenario B: Batch Exporting

Definition: Multiple RAW files are processed a single time at full resolution and written to disk as JPEG or HEIF.

• Disable Intermediate Caching: Turn cacheIntermediates to false on your export CIContext. Since each image is rendered only once, caching intermediates provides no speed benefit and wastes massive amounts of memory.
• Raise the Context Memory Limit: iOS defaults to a conservative 256 MB memory limit per context. Raise this to 512 MB or 1024 MB via CIContextOption.memoryLimit to drastically accelerate full-resolution exports.
• Use Native Representation Exporters: Avoid routing through ImageIO directly. Use native CIContext methods like heifRepresentation(of:format:colorSpace:options:) or jpegRepresentation(of:format:colorSpace:options:) for optimized, low-overhead memory usage.

4. Advanced CIImageProcessor Enhancements

For developers building custom image pipelines, Apple has exposed two powerful features within CIImageProcessor that were designed to facilitate RAW 9's machine learning backend.

Feature 1: Explicit Output Tile Sizes

By default, Core Image handles image subdivision implicitly based on system memory constraints. However, machine learning frameworks like Core ML require exact, predictable input boundaries (e.g., matching a model's expected fixed tensor dimension).

You can now explicitly dictate custom tiling geometry during the processor invocation:

Figure 2: Breaking an image space down into explicit 512x512 grids

// Divide the image size space into explicit 512x512 processing regions
let imageExtent = inputImage.extent
let tileSize = CGSize(width: 512, height: 512)
var tiles: [CGRect] = []

// Calculate rects across the image canvas
for y in stride(from: imageExtent.minY, to: imageExtent.maxY, by: tileSize.height) {
    for x in stride(from: imageExtent.minX, to: imageExtent.maxX, by: tileSize.width) {
        let tileRect = CGRect(x: x, y: y, width: tileSize.width, height: tileSize.height).intersection(imageExtent)
        tiles.append(tileRect)
    }
}

// Pass explicit tiles directly into your custom processor call
let outputImage = MyCustomProcessor.apply(withExtent: imageExtent, inputs: [inputImage], arguments: [kCIImageProcessorOutputTileKey: tiles])

Feature 2: Recycled Temporary Buffers

Core Image native inputs typically use interleaved color channel buffers (RGBA). Core ML models require planar data (separate layers of R, G, and B arrays).

Converting between these formats across thousands of image tiles requires temporary scratch buffers. Creating and destroying these buffers repeatedly introduces memory allocation overhead. You can leverage the automated buffer pool recycling via CIImageProcessorOutput:

class MyCustomProcessor: CIImageProcessorKernel {
    override class func process(with inputs: [CIImageProcessorInput]?, output: CIImageProcessorOutput) throws {
        guard let input = inputs?.first else { return }
        
        // Request a recycled, temporary CoreVideo pixel buffer using a specific key
        let bufferIdentifier = "PlanarConversionScratchSpace" as CFString
        
        guard let scratchBuffer = output.typedTemporaryBuffer(
            withWidth: Int(output.region.width),
            height: Int(output.region.height),
            pixelFormat: kCVPixelFormatType_32BGRA,
            identifier: bufferIdentifier
        ) else { return }
        
        // 1. Copy from input.pixelBuffer to scratchBuffer
        // 2. Perform your specialized processing/CoreML inference inside scratchBuffer
        // 3. Write results from scratchBuffer over to output.pixelBuffer
        
        // Core Image automatically manages the lifecycle of this scratchBuffer,
        // recycling it for subsequent tiles instead of deallocating it.
    }
}