How do I use an image from ImageCropper with Windows.Media.Ocr?

Question

I am attempting to use an image from the ImageCropper in Windows Community Toolkit (https://learn.microsoft.com/en-us/windows/communitytoolkit/controls/imagecropper) with Windows.Media.Ocr. This is my first time using either one of these, and my OcrResult object claims the image has no text (the Text property has a value of ""). My app displays the ImageCropper with it's image with no problem, so the original image is definitely there, but I cannot seem to get the cropped image, therefore preventing me from using it in the Ocr. I'm sure this code is very simple once I see it, but right now, I'm struggling to find it. Can anybody help me? Thanks.

Answer 1

Here is the code I am using to get an image from the camera (cePuzzle is a CaptureElement & puzzlecapture is a MediaCapture & icPuzzle is an ImageCropper):

Private Async Sub btnStartStopScanning_Click(sender As Object, e As RoutedEventArgs) Handles btnStartStopScanning.Click
Select Case Me.puzzlecapture.CameraStreamState
Case CameraStreamState.NotStreaming
Me.btnStartStopScanning.Content = "Stop Scanning"
Me.btnLookForPuzzle.IsEnabled = False
Me.cePuzzle.Visibility = Visibility.Visible
Me.icPuzzle.Reset()
Await Me.puzzlecapture.StartPreviewAsync()
Case CameraStreamState.Streaming
Me.btnStartStopScanning.Content = "Start Scanning"
Me.btnLookForPuzzle.IsEnabled = True
Me.cePuzzle.Visibility = Visibility.Collapsed
Await Me.puzzlecapture.StopPreviewAsync()

'Create a temporary file for the ImageCropper from the camera
Dim sf As StorageFile = Await ApplicationData.Current.LocalFolder.CreateFileAsync("temppuzzleimage.jpeg", CreationCollisionOption.ReplaceExisting).AsTask()
'Capture the image from the camera
Await Me.puzzlecapture.CapturePhotoToStorageFileAsync(ImageEncodingProperties.CreateJpeg(), sf)
'Place the image in the ImageCropper
Await Me.icPuzzle.LoadImageFromFile(sf)
'Case CameraStreamState.BlockedForPrivacy
'Case CameraStreamState.Shutdown
End Select
End Sub

This code works fine, it takes the image & puts it in the ImageCropper, so I can see it & it lets me select a region to crop. However, when I use the following code (which may be incomplete * the OCR is currently commented out, since I have been trying things which were obviously incorrect) it does not seem to work (there is no code there to create the SoftwareBitmap because I was probably doing it wrong as well, so most of this code probably needs rewritten or is missing):

Private Async Sub btnLookForPuzzle_Click(sender As Object, e As RoutedEventArgs) Handles btnLookForPuzzle.Click
Dim sf As StorageFile = Await ApplicationData.Current.LocalFolder.CreateFileAsync("tempcroppedimage.png", CreationCollisionOption.ReplaceExisting)
Dim stream As Streams.IRandomAccessStream = Await sf.OpenAsync(FileAccessMode.ReadWrite)
Await Me.icPuzzle.SaveAsync(stream, BitmapFileFormat.Jpeg)
'Dim bmp As SoftwareBitmap

'Dim ocr As OcrEngine = OcrEngine.TryCreateFromUserProfileLanguages()
'Dim result As OcrResult = Await ocr.RecognizeAsync(bmp)
'For Each textline As OcrLine In result.Lines
' System.Diagnostics.Debug.WriteLine(textline.Text)
'Next
'System.Diagnostics.Debug.WriteLine(result.Text)
End Sub

Everything I have tried would give an OcrResult that has found no results (result.Text is ""). What should I be doing? Thanks.

Answer 2

I tried changing the second method (btnLookForPuzzle) to the following:

Private Async Sub btnLookForPuzzle_Click(sender As Object, e As RoutedEventArgs) Handles btnLookForPuzzle.Click
    Dim sf As StorageFile = Await ApplicationData.Current.LocalFolder.GetFileAsync("temppuzzleimage.png")
    Dim stream As Streams.IRandomAccessStream = Await sf.OpenAsync(FileAccessMode.ReadWrite)
    Await Me.icPuzzle.SaveAsync(Stream, BitmapFileFormat.Png)
    Dim decoder As BitmapDecoder = Await BitmapDecoder.CreateAsync(stream)
    Dim bmp As SoftwareBitmap = Await decoder.GetSoftwareBitmapAsync()
    Dim ocr As OcrEngine = OcrEngine.TryCreateFromUserProfileLanguages()
    Dim result As OcrResult = Await ocr.RecognizeAsync(bmp)
    For Each textline As OcrLine In result.Lines
        System.Diagnostics.Debug.WriteLine(textline.Text)
    Next
    System.Diagnostics.Debug.WriteLine(result.Text)
End Sub

This finds some little pieces of text, but not nearly everything, and a large amount of what it does find is incorrect. Most of what it should find is single digits, almost none of which it finds, and some of the text it finds incorrectly. The text is solid black on a white & yellow background, so the contrast is very distinct, and the font is very basic (it is not some fancy script or anything like that). However, this also gives the following exception the second time the method is called:

Exception thrown: 'System.IO.FileLoadException' in System.Private.CoreLib.dll
WinRT information: The file is in use. Please close the file before continuing.

I'm sure that this is also something that can be easily fixed, but since I am not experienced with working with files, I'm not sure how to fix it. How do I fix this exception, and why is the OCR having so much trouble finding basic text? Thanks.

Answer 3

Here is one of the images I tried scanning (Note: The bottom of the image I am sending is cropped off because of the size limit, but the part that was cropped off is the same as the part near the bottom of the image):

The text that I was expecting it to find would be similar to the following:

EASY
VOLUME 256
6 1 4 5

In place of "EASY" it found several seemingly random characters that didn't even look similar to me, in most cases, it did find "VOLUME 256", and it never found anything else (the numbers, which are actually the part that I care about it finding). I found it very surprising that it would find "VOLUME 256" but not "EASY", since it looks like they are the same (or at least a similar) font, and "EASY" is actually a little bit larger. The numbers are a little bit thinner, so that is my best guess as to not finding them, although they still look pretty clear. Any ideas?