Getting editible text from a photo with Shortcuts on iOS with OCR

Written on November 15, 2020

Note taking while reading a physical book is one of the best ways for a concept to stick. But what if you don’t want to write in your book and rather take an excerpt from the book and save it in a digital note taking app like Notes on iOS?

You could type out the section you want to quote in your Notes app but this could take a bit of effort.

But what if you could take a photo of the excerpt you want to record in your Notes app and then have iOS grab the text from the photo for you to paste in your note? This would be so much more simpler and faster.

This would also allow you to get to capturing your own thoughts around the part of the book that you’re reading quicker.

With Shortcuts on iOS and an app called Toolbox Pro—which adds even more actions in Shortcuts—you can rely on iOS’s on-device optical character recognition (OCR) capability to achieve this better workflow for recording book notes.

In this article, I’ll show you how to put together this shortcut in Shortcuts. But before diving in, here’s a video of what the shortcut looks like in action.

Before we get started

Before we get started in setting up the shortcut in Shortcuts, you’ll need to make sure you have a copy of Toolbox Pro. Otherwise you won’t be able to access the OCR action in Shortcuts.

You’ll also need to pay the in-app purchase for Toolbox Pro Premium as the OCR action is a premium action.

The price you pay for premium is worth it alone for the ability to have on-device OCR using Shortcuts as it will save you a ton of time from typing. Not to mention, if you’re an iOS automation lover, the premium set of actions opens up for so many more powerful capabilities in Shortcuts.

Setting up a shortcut in Shortcuts to OCR notes from photo

This OCR shortcut can be created using just five actions.

  1. The first action you’ll want to add to your shortcut is the Take Photo action. You’ll want to set the action to use the Back camera to capture the text from your book.

    I also like to toggle on the Show Camera Preview option—found under Show More—for this action so that I can see what exactly the camera grabbed.

    Camera
    Take 1 photo with Back camera
    Show Camera Preview
  2. The next action you’ll want to add is the Crop Image Manually action from Toolbox Pro. What this action will do is take the photo you took from the action above and then provide you with the ability to crop the photo. This is important because you can fine tune which bit of text the OCR captures.

    You’ll also want Toolbox Pro to pass back to Shortcuts to continue this shortcut, so you’ll want to toggle on the Open URL when finished? option.

    Icon for Toolbox Pro
    Toolbox Pro
    Crop Image Manually
    Image
    Photo
    Open URL when finished?
    Post URL
    shortcuts://
  3. Next you’ll add the Wait to Return action. What this will do is pause the shortcut from running further until Shortcuts is switched back into focus. This is important because without it, your shortcut will continue to run and it is important that each action finishes processing before proceeding to the next step.

    Scripting
    Wait to return
  4. What the action in step 2 does is takes the cropped image and makes it available in iOS’s clipboard. For this reason, we now want to take the content of the clipboard by placing a Get Clipboard action next.

    Sharing
    Get clipboard
  5. Now we’ll pass this image to Toolbox Pro once more to actually do the OCR by passing it to the action Get Text From Images action. This will cause Toolbox Pro to open and the OCR magic will occur here. Once the OCR is complete, the new text is now made available in iOS’s clipboard.

    Icon for Toolbox Pro
    Toolbox Pro
    Recognise text in Clipboard
    Correct Language Errors
    Open URL when finished
    URL
    shortcuts://
    Language
    Choose

And that’s it. Now you have the ability to take a photo of a page from a book and get the editable text in your clipboard to paste it into any note taking app in a matter of minutes.

What the entire shortcut looks like with comments

Here’s a representation of what the entire shortcut looks like in Shortcuts with comments.

Comment

First grab the photo with the desired text, then pass it to Toolbox Pro to manually crop the photo.

The cropped photo is then saved to the clipboard.

Camera
Take 1 photo with Back camera
Show Camera Preview
Icon for Toolbox Pro
Toolbox Pro
Crop Image Manually
Image
Photo
Open URL when finished?
Post URL
shortcuts://
Scripting
Wait to return
Comment

Grab the photo from the clipboard and then pass it to Toolbox Pro again to recognize the text from the image.

After the text is grabbed from the photo it gets saved to the clipboard, ready for you to paste in whichever note taking app you desire.

Sharing
Get clipboard
Icon for Toolbox Pro
Toolbox Pro
Recognise text in Clipboard
Correct Language Errors
Open URL when finished
URL
shortcuts://
Language
Choose

Grab copy of the OCR shortcut

If you’d like, here’s a link to download a copy of the shortcut on your own device.

If you have any questions or comments about how to improve this shortcut, please reach out on Twitter.

Stay in touch

Thanks for reading this article. I'd love to stay in touch and share articles like this one in your inbox. Sign up for my newsletter.