question

FischerMatthias-1742 avatar image
0 Votes"
FischerMatthias-1742 asked LyndseySandersonCynetixGroup-4120 commented

Powershell Invoke-WebRequest to https://docs.microsoft.com

Hi guys,
we tried to invoke a webrequest with powershell to https://docs.microsoft.com (https://docs.microsoft.com/en-us/officeupdates/update-history-microsoft365-apps-by-date) to get the versionnumber table for exchange via ps1 script. It worked fine since today.
Now all invoke-webrequest to docs.microsoft.com without basic parsing seems to be blocked with powershell running into an endless loop without error. Anyone else got problems like this?
Thanks!

windows-server-powershell
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

AndreasBaumgarten avatar image
0 Votes"
AndreasBaumgarten answered FischerMatthias-1742 commented

Hi @FischerMatthias-1742 ,

here i tried the Invoke-WebRequest and got a response:

 Invoke-WebRequest -URI https://docs.microsoft.com/en-us/officeupdates/update-history-microsoft365-apps-by-date

81137-image.png




(If the reply was helpful please don't forget to upvote and/or accept as answer, thank you)

Regards
Andreas Baumgarten


image.png (299.8 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Neither me nor my collegues get a response from this site. Just this hanging shell without errorcode.
81165-image.png



0 Votes 0 ·
image.png (16.2 KiB)
RichMatheisen-8856 avatar image
0 Votes"
RichMatheisen-8856 answered

I get the same result . . . it just hangs. Doesn't respond to Ctrl+C or any other attempt to kill it.

Adding "-Verbose" to the cmdlet showed this:

PS C:\Users\richm> Invoke-WebRequest -URI https://docs.microsoft.com/en-us/officeupdates/update-history-microsoft365-apps-by-date -verbose
VERBOSE: GET https://docs.microsoft.com/en-us/officeupdates/update-history-microsoft365-apps-by-date with 0-byte payload
VERBOSE: received -1-byte response of content type text/html


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

RichMatheisen-8856 avatar image
0 Votes"
RichMatheisen-8856 answered

Interesting. Assigning the result of that Invoke-WebRequest to a variable works. You can access the RawContent, RawContentLength , Content, Headers, etc. However, the parsing of the data fails by entering a loop. E.g., $x.Links or $x.ParsedHTML never complete.

The BaseResponse property is this -- note the ContentLength value!

 IsMutuallyAuthenticated : False
 Cookies                 : {}
 Headers                 : {Request-Context, X-Datacenter, X-Frame-Options, x-content-type-options...}
 SupportsHeaders         : True
 ContentLength           : -1
 ContentEncoding         :
 ContentType             : text/html
 CharacterSet            : ISO-8859-1
 Server                  :
 LastModified            : 3/24/2021 10:52:46 AM
 StatusCode              : OK
 StatusDescription       : OK
 ProtocolVersion         : 1.1
 ResponseUri             : https://docs.microsoft.com/en-us/officeupdates/update-history-microsoft365-apps-by-date
 Method                  : GET
 IsFromCache             : False


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Paul-4124 avatar image
0 Votes"
Paul-4124 answered RichMatheisen-8856 commented

I am glad we're not alone in seeing this behavior. I will monitor and update this if I hear anything.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I expect that the problem lies in the HTML data. Submitting that page to an HTML verifier results in many errors. Have a look:
validator.w3.org
html-validator.html
html5.validator.nu

html-validator


0 Votes 0 ·
Paul-4124 avatar image
0 Votes"
Paul-4124 answered

Guys, I can't take the credit (although we did punt it to MS as a question if you want to give me kudos!) but it appears that MS have fixed this issue.

Good news indeed.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Paul-4124 avatar image
0 Votes"
Paul-4124 answered Paul-4124 commented

Anyone noticed that IWR seems to not work again?

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Same for me, checked it right now and it doesnt work again.

1 Vote 1 ·
Paul-4124 avatar image Paul-4124 FischerMatthias-1742 ·

Thank you, I'll raise a ticket and see if I get anywhere

0 Votes 0 ·
AntnioPinho-1406 avatar image
0 Votes"
AntnioPinho-1406 answered
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Paul-4124 avatar image
0 Votes"
Paul-4124 answered LyndseySandersonCynetixGroup-4120 commented

I've just gotten a resolution from Microsoft and it mirrors what is seen in @AntnioPinho-1406 's post.

Add a DWORD value of "PowerShell.exe" and/or "PowerShell_ISE.exe" with a value of "00002af9" to the following location. You must restart for this to take effect.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_BROWSER_EMULATION

or

HKEY_CURRENT_USER\SOFTWARE\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_BROWSER_EMULATION

I am guessing that @AntnioPinho-1406 is from Microsoft as in the example the same URL that I supplied as a problem in the ticket is used!.

This will work with either of these examples:

1 - Without UseBasicParsing enabled


Invoke-WebRequest -Uri "https://docs.microsoft.com/en-us/windows-server/remote/remote-desktop-services/clients/windowsdesktop-whatsnew"


2 - with BasicParsing (which may come in handy if PowerShell 7 becomes the norm


         $page = Invoke-WebRequest -Uri "https://docs.microsoft.com/en-us/windows-server/remote/remote-desktop-services/clients/windowsdesktop-whatsnew" -UseBasicParsing
         # Create HTML file Object
         $HTML = New-Object -Com "HTMLFile"
         # Write HTML content according to DOM Level2 
         $HTML.IHTMLDocument2_write($page.Content)
         $HTML.getElementsByTagName('h2') | select id | Where{ $_.id -match "updates-for-version" -and $_.id -notmatch "insider"} | select -first 1
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

The registry key worked for me, thank you for this.
Lyndsey

0 Votes 0 ·
AntnioPinho-1406 avatar image
0 Votes"
AntnioPinho-1406 answered Paul-4124 commented

That's correct Paul, I've investigated this issue late last week and my findings/testing are included in the mentioned GitHub issue. Even though this will be fixed at the site level any time soon (see https://github.com/microsoft/DynamicProto-JS/issues/31), we'll be working internally to see how can we improve this scenario so we hopefully do not hit this issue type again.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Fantastic AntnioPinho, I really appreciate this.

I was wondering in PowerShell 7 I believe that UseBasicParsing is the default option when invoking a WebRequest. Is there a way to override that and still use "rich parsing" as it were?

0 Votes 0 ·
Paul-4124 avatar image
0 Votes"
Paul-4124 answered

This is now resolved! Had confirmation from MS and I've tested myself.

You DO NOT need the additional keys defined in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_BROWSER_EMULATION

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.