Edit desktop UI flows

Desktop UI flows automate Windows desktop applications. Please refer to the Known issues to learn more about issues you might run into, workarounds for those issues, and scenarios that are not supported in this release.

Prerequisites

A desktop UI flow. Create a desktop UI flow now if you don't have one to edit.

Edit actions

Edit actions

You can edit your recording to:

  • Modify the value for actions that support it.
  • Delete a step.
  • Delete the recording.
  • Change the order of actions with drag and drop. Do exercise caution with this as it may break the consistency of your recording.

Advanced parameters let you change:

  • The delay after the action is performed. For example, you can add a one second delay by changing PT0S to PT1S. This can be useful when the target application has a slow response time that doesn’t complete before the next step of your UI flow. You can also add delay actions explicitly so that they can be used in loops, for example.
  • The selector for the target user interface element.

Add a recording

You may want to record your UI flow in multiple sessions. After you've completed your first recording you can proceed as follows:

  1. Sign into Power Automate.

  2. Select My flows > UI flows.

  3. Select the UI flow that you want to edit.

    Image that displays the UI flow to be edited

  4. Select Edit.

  5. Select New step.

    New step

  6. Select Record app from the list of actions.

    Record app

  7. Select Launch recorder.

    Select launch recorder

    The recorder control displays at the top of your screen.

    The recorder-control

  8. Start the app that you want to record.

    Tip

    As your mouse hovers over controls in the app, you'll notice that a blue outline highlights each control. Always wait for the blue highlight before you select a control.

    If the blue highlight doesn't display around the element, it might not be recorded properly.

  9. Select Record from the recorder control.

  10. Perform the steps in the user interface of the app you are recording, and then select Done on the recorder control.

  11. Select Save, and then test your UI flow.\

Provide command line arguments at launch

After you record your UI flows actions, you can provide command line arguments for the apps that your UI flows launch.

Follow these steps to add command line arguments to the apps that your UI flow launches:

  1. Record a UI flow with the desktop recorder.

  2. Select Done on the recorder, select the application launch action to expand it.

    The screenshot uses Notepad++ as an example.

    Displays the launch application step

  3. Select Show advanced options.

    Select the link to reveal the advanced options

  4. Provide the command line arguments here:

    Displays the text field for the command line arguments

  5. You can provide any valid command line argument for your application.

Important

Do not pass sensitive text like passwords through command line arguments.

Add a variable

You can use variables to perform multiple actions within your UI flows. For example, you can use a variable to count how many times an operation was performed in a loop. Or, you can use a variable to create an string array with the screen readings done in a loop.

Tip

You can use integer, float, Boolean, string, array, and object variable data types.

After you create a variable, you can perform other tasks, for example:

  • Increase or decrease the variable by a constant value, also known as increment and decrement.
  • Insert or append the variable’s value as the last item in a string or array.
  • Assign a different value to a variable.

Follow these steps to create and use variables in a UI flow:

  1. Go to the beginning of the UI flow and select Add an action.

    Select add an action

  2. Search for the word variable, and then select the Initialize variable action under the built-in category.

    Select the initialize variable action

  3. Initialize the variable to type that meets your needs.

    Initialize the variable to the type you need

Tip

If your UI automation scenario involves reading values from a screen in a loop, you could achieve this by initializing the variable as an array type.

Then, within the UI flow, select the Append to Array variable action, and then use the output of Get Text action from dynamic content list.

You can append to an array in a loop

Copy and paste steps

You can copy actions and scopes, use Add an action to select the copied actions and scopes from My clipboard, and then paste them into another location within the same UI flow.

UI flows automatically renames the copied actions and scopes with a unique name. If the copied scope contains another scope, the nested structure is retained when you paste it from My clipboard.

Important

This feature supports copying and adding actions within the same scope. You cannot paste actions into other scopes or UI flows.

Follow these steps to copy and then paste actions and scopes:

  1. Select , and then select Copy to my clipboard for the action or scope that you want to copy.

    View of the copy clipboard

  2. Hover over the target location, and then select Add an action.

    Tip

    You can select Add an action directly if it is at the end of the scope or UI flow.

    Add an action selected

  3. Select My clipboard, and then select the action or scope you copied.

    Select the copied action

  4. Notice that UI flows pastes the action card with a unique name.

    View the new name of the copied action

  5. Also notice that any actions with screenshots will also retain them for easy reference.

    View retained screenshots

    More retained screenshots

Paste a scope immediately after itself

Note

If a scope does not have a Close Application action, and the user copies the scope and then pastes the scope immediately after the existing scope, the pasted actions will only be performed within the first application instance.

Paste a scope

In this example, the highlighted actions Left click 11 and Replay keystrokes will be performed in the first Notepad application, and another notepad instance will be launched.

To address this instance issue, users have two options:

  1. If you only want to perform the action(s) in the first application instance, you can copy and paste only the relevant actions in the first scope. If you want to copy and paste the entire scope at once, you should remove the Launch action in the pasted scope to prevent launching an extraneous application instance.

    Relevant actions only

  2. If you want to perform the action(s) in a separate application instance, you can add a Close application action in the first scope, so that the second scope will have its own instance.

    Separate application instance

Add a loop

UI flows have support for arrays. Use an array, to add a loop to perform repetitive steps in a UI flow. You can pass in different types of arrays from a flow to your UI flow, including complex arrays from a SharePoint list, for example.

Note

You must edit an existing UI flow to add a loop.

Here are the steps to create a loop in a UI flow:

  1. Edit the UI flow to which you want to add the loop.

    Select the edit button for the UI flow

  2. Navigate to input section of the wizard, and then select Set up inputs.

    Select Set up inputs

  3. Select Array to indicate that you want to create an array input type.

    Select Array

  4. Enter the details for the array including a name, the array in JSON format, and a description for the array.

    Details of the array

    Tip

    If you are using a complex array from a SharePoint list, you don't have to create the array manually. You can copy the array from a flow and paste it into the middle box.

  5. Select Next.

  6. Select Add an action .

  7. Search for "to each" > Built-in > and then select Apply to each.

    Select the apply to each action

  8. Select Select an output from previous steps, select the array you defined earlier from the Dynamic content.

    Add the array from dynamic content

  9. Drag the actions you want to repeat and drop them onto the Apply to each card.

    For example, I've dropped the LeftClick 4 onto the Apply to each card in the following image.

    Drag the step you want to repeat onto the array from dynamic content

  10. Replace the input tokens created during the recording with the array input values as needed.

Tip

If the array is a simple string or integer array, you can use the Current item directly from the Dynamic content, like the following image. You can also use expressions.

Use Current item

Known Limitations:

  1. Loops require static selectors. This means that the user interface (UX) element on which the repetitive action happens must not change.
  2. Output operations are not currently supported.
  3. For complex array objects that need to be passed from a flow, like a Sharepoint list, provide an example payload of that object. To get the payload, you can run the flow without the UI flow step, and then copy the output of the action into the middle (Add sample data) text box when you define the array.
  4. You must use expressions for complex array object types. For example, use the expression items(‘Apply to each’)[‘’], where is name of the specific object within the input array.

Add a delay

You can add delay in UI flows to help control the UI flow run.

Follow these steps to add a delay to a UI flow that you've recorded.

  1. In the desired spot, hover over the arrow and select Insert a new step. Alternatively, for adding it to the end of the scope or UI flow, select Add an action directly.

    Insert a delay step

  2. Select Add an action.

    Add the delay action

  3. Select Built-in, and search for "delay". Alternatively, select Schedule > Delay.

    Search for the delay action

  4. Enter the count, for example "2" to represent the number of the Unit.

  5. Select the Unit, for example minutes, to represent the length of time for the delay.

    The following image shows a delay of two minutes.

    Define the delay

    The delay action is added to the UI flow. Now, when the flow runs, there will be a delay as you've defined before the next action runs.

Add a retry policy

By default, UI flows retries failed UI automation steps using the default retry settings and timeout value.

Follow these steps to set your own retry policy on specific steps after you've recorded your UI flow.

  1. Select ... on the step that you want to change the retry policy.

  2. Select Settings.

    The settings card for the step opens.

    Open settings card

  3. Add a timeout value in the Duration box.

    The timeout duration can be between one minute and 24 hours. For example, you must enter the timeout duration in this format: PT1M to represent a timeout duration of one minute or PT24H to represent a timeout of 24 hours.

    Retry policy card

    The timeout is a value you can specify to set the length of time the UI flow should try to perform an action that it’s unable to complete before canceling it. Once the action is canceled, the retry policy begins. The default timeout value is five minutes.

    There are three types of retries in UI flows.

    • Default
    • None
    • Custom

The Default option is selected by default. This default value is set to retry 9 times. Including the initial attempt, this means a total of 10 attempts. There is a 1 second interval between retries.

If you select the None option, then no retry attempts are done for this step.

If you want to specify a custom retry policy, then select the Custom option, and enter value for Number of retries and Length of intervals.                   

Note

The retry policy isn't available for all steps. If you don't see the retry options, it means that the policy isn't available for the specific step.

Add a manual action

Once you've recorded an application with at least one action, you can manually add any of the following actions for that application.

Action Comment
Close application
Right click
Send keys Send keys and key combinations, such as CTRL + C.
Left click
Get text Read the text from a user interface element and then use it as an output.
Enter text
Get element enabled Check if a user interface element is enabled or disabled.
Clear element Clear the value in an editable user interface element.
Wait for seconds Wait before continuing to the next step.

Follow these steps to add a manual action:

  1. Sign into Power Automate.

  2. Select My flows > UI flows.

  3. Select the UI flow that you want to edit.

    The UI flow to edit

  4. Select Edit.

  5. Select the recording card that contains the steps to which you want to add a new step. The card expands and displays the recorded steps.

    Select recording card

  6. Select Add an action on the recording card, just below the last recorded step. You'll see the list of manual actions listed earlier in the walkthrough.

  7. Select the action you want to add. Here, I've selected Get element enabled, but you can select any action that makes sense for your scenario.

    Select action to add.png

Once the action is added, you will need to set the Selector in the action’s advanced options.

Action advanced options

Set the selector

The selector identifies the user interface element onto which the action is performed during playback. We recommend that you copy/paste this information from a separate step targeting the same user interface element, if possible.

The format of the selector is:

{  
   "type":"WinUIA",
   "parameters":{  
      "elementStack":[  

      ],
      "elementXPath":""
   }
}

You need to provide the data for the the elementStack and elementXPath fields of the selector element.

Here is an example of what the elementStack might look like.

Element stack

You can capture the elementXPath using the WinAppDriver UI Recorder.

WAD tool

Remove the first element (everything before /Window) before using the result in elementXPath of the selector.

Test your UI flow to confirm that your selector works correctly.

Use advanced controls

You can incorporate advanced controls such as conditions, switch cases, and terminate actions within your UI flows.

You can perform these advanced operations by performing the following steps on an existing UI flow.

  1. Select the UI flow you have created > More commands (...) > Edit.

  2. Select + > Add an action on the down arrow of UI flow step to which you want to add logic.

    Add an action

  3. Select Built-in, and then select one of the control actions available.

    Built-in

  4. Complete the expression that needs to be evaluated. You may use dynamic content and expressions to evaluate your condition and switch controls. Additionally, you can use any output that's generated from the previous UI flow steps.

    Condition card

Add a recording of a remote computer using image recognition (Preview)

We expect changes to this feature, so you shouldn’t use it in production. Use it only in test and development environments.

Image recognition in UI flows is a preview feature and currently available when recording a remote computer through the Remote Desktop Connection (RDC) app.  

What is image recognition?

Currently, desktop UI flows are primarily recorded using accessibility APIs (UI Automation and WinAppDriver) to detect the controls in the Microsoft Windows UI tree.​ Sometimes, the UI tree isn't available, such as with Web-based or Java apps. The UI tree may may be unreliable, such as when the IDs of a control change frequently or between sessions.

With image recognition, click locations and other details are visually matched during playback, greatly expanding the range of applications that can be automated.

Use image recognition to record a remote computer

  1. In a new or existing UI flow, go to the Inputs tab and create two new sensitive text inputs, one for the username and one for the password that will be used to log in to the remote device. Sensitive text inputs allow you to pass in the values dynamically when testing or calling the UI flow from another flow, without them being stored or logged by the UI flow itself.

    Sensitive text

  2. Follow the steps in Add a recording to launch the recorder control for a new or existing UI flow.

  3. Use the Remote Desktop app to connect to the remote computer.

  4. Expand the Remote Desktop window to full screen.

  5. Select Record from the recorder control and select Got it in the alert that appears.

    Popup

  6. Perform the steps on the remote computer, and then select Done on the recorder control.

  7. Locate the Launch Remote Desktop action within your recording, and then enter the sensitive text inputs for the username and password.

Username and password sensitive text

  1. Select Save, and then test your UI flow.

Important

When you call this UI flow from an automated flow, it is recommended to use a key management solution such as Azure Key Vault to retrieve the username and password and then pass them to the sensitive text inputs in the UI flow dynamically, rather than storing them in the flow itself.

Tip

Enable Secure Inputs in the Settings menu for the Run a UI flow for desktop action in the flow that is calling the UI flow. This ensures that the inputs are not stored in the run history.

Use image recognition to record on Citrix

There are two options to record on Citrix with UI flows.

Option 1: record a Citrix application or desktop already open on your machine

  1. Follow the steps in Add a recording to launch the recorder control for a new or existing UI flow.
  2. Use the Citrix app or remote desktop to connect to the remote computer.

View of the Citrix app

  1. If you are using Citrix remote desktop, expand the window to full screen.
  2. Select Record from the recorder control, and then select Got it in the alert that appears.
  3. Perform the steps on Citrix, and then select Done on the recorder control.

Tip

UI flows, you can record several Citrix apps during the same session. Open all applications before you start recording if you have not recorded the launch of your Citrix workspace.

Option 2: record the launch of your Citrix application / desktop from your Citrix workspace

  1. Follow the steps in Add a recording to launch the recorder control for a new or existing UI flow.

  2. Open the Citrix workspace from the taskbar.

  3. Enter your password, and then and select Log on.

    Tip

    If you don’t want to select Remember my password, use a Sensitive text input.

    Enter the Citrix app password

  4. Select the App or the Desktop you want to record.

    Shows a list of apps to select to record

  5. Perform the steps on Citrix, and then select Done on the recorder control.

Important

If you recorded the log-on step, the designer displays three Citrix scope applications. This is expected because the log-on window is a different application from the Citrix workspace.

Tip

In test and attended mode, confirm that your Citrix workspace is closed before you playback your UI flow. From the notification bar, right-click on the Citrix icon and the select Exit to close the workspace.

Troubleshooting

If you have problems using sensitive text in the login window:

  1. Open the input menu from the recorder.

  2. Select the password field.

  3. Select your sensitive text (in this example “password”) in the input menu.

    Use sensitive text for the password

Use OCR to extract text from images

  1. While recording your steps, navigate to the location of the text that you wish to capture.

  2. Select Outputs > Extract Text from Image from the recorder control.

  3. Follow the prompts to select an anchor area (a section of the screen that is not expected to change, such as the label next to a field).

    Anchor area

  4. Select the target area (the area from which text will be extracted using OCR).

    Target area

  5. Enter a name for the output.

  6. Select Done on the recorder control.

  7. Select Save, and then test your UI flow.

Known issues for Remote Desktop recordings

  1. Ensure that all required inputs (computer name, username, and password) are filled in and saved before recording further steps in the same UI flow.

  2. To attach to an existing remote desktop session, the session must have been launched earlier in the same UI flow.

  3. The recommended way to launch the Remote Desktop Connection (RDC) when recording is from the Remote Desktop Connection app (mstc.exe) from the Start menu. If Windows Security actions are recorded in addition to the Launch Remote Desktop action, they should be removed from the designer so that playback is not interrupted (this can happen when the Remote Desktop session is launched from a shortcut).

  4. Playback can fail if the UI flow was recorded on a screen with screen scaling (Windows Settings > Display > Screen Scaling) set to a value other than 100%. As a workaround, ensure that screen scaling is set to 100% prior to recording.

Handle error conditions

Unexpected conditions can arise during playback. These conditions can cause your UI flows to fail. You can use advanced error handling capabilities to create alternate steps when unexpected conditions arise.

Here are the steps to do so.

  1. Sign into Power Automate with your school or work account.

  2. Select My flows > UI flows.

  3. Select More commands (the three vertical dots for the UI flow you want to edit).

  4. Select Edit.

  5. Select the down arrow that's located immediately before the UI flow step to which you want to add error handling, and then select + (Insert new step).

    In the following image, we're inserting the new step before the PostElementText 1 step. This means that if PostElementText 1 fails, the alternate steps that you define next, run.

    Insert new step image

  6. Select Add a parallel branch.

    Image showing add a parallel branch and other options

  7. Select the action that you want to perform in the parallel branch if there's a failure when the UI flow runs.

    You can select Record app to do a new recording for the parallel branch or select Terminate from the Built-in actions to exit the UI flow exit gracefully if there's a failure.

    Image showing the parallel branch options

    Note

    By default, the action that you add in the parallel branch runs only if the previous step fails. You can select ... on the parallel branch > Configure run after to change the default behavior.

    Image showing the options, including configure run after

  8. From this screen, you can select the condition on which you want the parallel branch to execute. You can select from one of the four options available.

    Image showing the run after options

    Note: You will not be able to save a UI flow that has both the main branch and parallel branch set to execute on the same condition.

Enable coordinate-based playback

Coordinate-based playback uses relative screen coordinate offset as a fallback that helps UI flows find target objects that the default Windows UI automation framework is unable to find during playback.

Here are some of the reasons why the default Windows UI automation framework might not find target objects during playback:

  • The legacy application that you're automating might not use programming technologies that support the Windows UI automation framework.
  • The application or its controls may not have a unique UI automation XPath, name, or IDs.
  • The application has dynamic controls whose names or IDs could change.
  • The application has controls that do not have names, IDs, unique identifiers, etc.

Tip

Use the same scale and resolution and also maximize the target application during recording to improve the accuracy of coordinate-based playback.

Follow these steps after you've recorded a UI flow script:

  1. Expand the step that launches or attaches the application.

    This is usually the first step of the recording script.

  2. Select Show advanced options.

  3. Find Use coordinate playback properties.

  4. Select Yes from the list to enable coordinate-based playback.

Tip

You can enable or disable Use coordinate playback for each application to apply the setting to all steps performed on that application.

Warning

With coordinate-based playback, the automation may select controls that are not part of the target application due to a variety of reasons, for example, when the UI of the target applications changes dramatically.

Next steps