question

Elizza avatar image
0 Votes"
Elizza asked Castorix31 commented

UIAutomation not catching all elements

Im using the code above to walk through the descendants of an element, however its not catching everything like i can see using inspect.exe.
inspect-objects

I'm testing in the chrome browser, in this current webpage, its catching just current opened pages, bookmarks, and browser buttons, its missing to catch the page content.

While testing in others windows like a folder for example, it does catch everything correctly, but its taking too much time, in a folder with 120 elements it takes up to 1.4 seconds, i wonder if the code could be improved in any manner?


 WCHAR window[250] = L"Ask a question - Microsoft Q&A";
 IUIAutomationElement *element = GetTopLevelWindowByName(window);
 ListDescendants(element, 2);
    
 IUIAutomationElement* GetTopLevelWindowByName(LPWSTR windowName)
 {
     if (windowName == NULL)
         return NULL;
    
     CComPtr<IUIAutomation> g_pAutomation;
     if (SUCCEEDED(CoInitialize(NULL)))
     {
         if (!SUCCEEDED(g_pAutomation.CoCreateInstance(CLSID_CUIAutomation8))) // or CLSID_CUIAutomation
             return NULL;
     }
    
     VARIANT varProp;
     varProp.vt = VT_BSTR;
     varProp.bstrVal = SysAllocString(windowName);
     if (varProp.bstrVal == NULL)
         return NULL;
    
     IUIAutomationElement* pRoot        = NULL;
     IUIAutomationElement* pFound       = NULL;
     IUIAutomationCondition* pCondition = NULL;
    
     // Get the desktop element. 
     HRESULT hr = g_pAutomation->GetRootElement(&pRoot);
     if (FAILED(hr) || pRoot == NULL)
         goto cleanup;
    
     // Get a top-level element by name, such as "Program Manager"
     hr = g_pAutomation->CreatePropertyCondition(UIA_NamePropertyId, varProp, &pCondition);
     if (FAILED(hr))
         goto cleanup;
    
     pRoot->FindFirst(TreeScope_Children, pCondition, &pFound);
    
 cleanup:
     if (pRoot != NULL)
         pRoot->Release();
    
     if (pCondition != NULL)
         pCondition->Release();
    
     VariantClear(&varProp);
     return pFound;
 }
    
    
    
 void ListDescendants(IUIAutomationElement* pParent, int indent)
 {
     static CComPtr<IUIAutomation> g_pAutomation;
    
     if (!g_pAutomation) {
         if (SUCCEEDED(CoInitialize(NULL)))
         {
             if (!SUCCEEDED(g_pAutomation.CoCreateInstance(CLSID_CUIAutomation8))) // or CLSID_CUIAutomation
                 return;
         }
     }
    
    
     if (pParent == NULL)
         return;
    
     IUIAutomationTreeWalker* pControlWalker = NULL;
     IUIAutomationElement* pNode = NULL;
    
     g_pAutomation->get_ControlViewWalker(&pControlWalker);
     if (pControlWalker == NULL)
         goto cleanup;
    
     pControlWalker->GetFirstChildElement(pParent, &pNode);
     if (pNode == NULL)
         goto cleanup;
    
     while (pNode)
     {
         BSTR sName;
         pNode->get_CurrentName(&sName);
    
         UIA_HWND uia_hwnd;
         pNode->get_CurrentNativeWindowHandle(&uia_hwnd);
    
         RECT rect;
         pNode->get_CurrentBoundingRectangle(&rect);
    
         ListDescendants(pNode, indent + 1);
         IUIAutomationElement* pNext;
         pControlWalker->GetNextSiblingElement(pNode, &pNext);
         pNode->Release();
         pNode = pNext;
     }
    
 cleanup:
     if (pControlWalker != NULL)
         pControlWalker->Release();
    
     if (pNode != NULL)
         pNode->Release();
    
     return;
 }



windows-apic++
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

XiaopoYang-MSFT avatar image
0 Votes"
XiaopoYang-MSFT answered XiaopoYang-MSFT edited

Perhaps page contents are not IUIAutomationElement nodes. It is a UI Automation document content client sample which can access the content of a document that is being displayed in another application's window.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Elizza avatar image
0 Votes"
Elizza answered XiaopoYang-MSFT commented

Hello XiaopoYang, the source code compiles only in vs2012? no chance to compile in vs19?

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

VS will retargets the project. Also have a look at Raw View.
130957-image.png

0 Votes 0 ·
image.png (19.2 KiB)
Castorix31 avatar image
0 Votes"
Castorix31 answered Castorix31 commented

You can get all the elements in the hierarchy with IUIAutomationElement::FindAll or IUIAutomationElement::FindAllBuildCache and TreeScope_Descendants
There are returned in an IUIAutomationElementArray array, so it is faster than IUIAutomationTreeWalker
If I test with Edge Chromium on this page, I get all the elements inside the page, like all your code

· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@Castorix31 could you share the code you used?

0 Votes 0 ·

Test with Edge Chromium : FindAllBuildCache
(EnumWindowsProc must be changed or replaced by FindWindow for other applications...)


0 Votes 0 ·

Here it only catches the page content if it's visible/active, there too?

0 Votes 0 ·
Show more comments