question

Valleze-6170 avatar image
0 Votes"
Valleze-6170 asked Valleze-6170 commented

How To Move Mouse Cursor to Specific Image based on Saved Image

I have a button image saved as png image.

I'm trying to create some autoclick in visual studio 2022 to click in this button ever it appears on my screen. But it appears in diferent positions and diferent times on screen.

I'm using c# language

I'm using winforms

saved button image is equal the real button image

I don't know about image recognition, but I know it's possible.

windows-forms
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

JackJJun-MSFT avatar image
0 Votes"
JackJJun-MSFT answered

@Valleze-6170, Welcome to Microsoft Q&A, based on my research, I find a solution to do it.

Please install following nuget-package:

187453-image.png

Then you could try the following code to move mouse cursor to button location.

Code:


  public int X { get; set; }
         public int Y { get; set; }
         protected override bool ProcessCmdKey(ref Message msg, Keys keyData)
         {
             if(keyData.ToString()=="K")
             {
    
                 Cursor.Position = new Point(X, Y);
                 return true;
             }
             return base.ProcessCmdKey(ref msg, keyData);
         }
         private void Form1_Load(object sender, EventArgs e)
         {
             Bitmap bitmap1 = new Bitmap(this.Width, this.Height);
             this.DrawToBitmap(bitmap1, this.ClientRectangle);
             bitmap1 = Example.ConvertToFormat((System.Drawing.Image)bitmap1, PixelFormat.Format24bppRgb);
             System.Drawing.Bitmap template = Example.ConvertToFormat(System.Drawing.Image.FromFile("1.png"), PixelFormat.Format24bppRgb);
             // create template matching algorithm's instance
             // (set similarity threshold to 90%)
             ExhaustiveTemplateMatching tm = new ExhaustiveTemplateMatching(0.9f);
             // find all matchings with specified above similarity
    
             TemplateMatch[] matchings = tm.ProcessImage(bitmap1, template);
             // highlight found matchings
    
             BitmapData data = bitmap1.LockBits(
                  new Rectangle(0, 0, bitmap1.Width, bitmap1.Height),
                  ImageLockMode.ReadWrite, bitmap1.PixelFormat);
             foreach (TemplateMatch m in matchings)
             {
    
                 Drawing.Rectangle(data, m.Rectangle, Color.White);
                 Console.WriteLine(m.Rectangle.Location.ToString());
                 // do something else with matching
                 X = m.Rectangle.X;
                 Y = m.Rectangle.Y;
             }
         }
    
  public static class Example
     {
         public static Bitmap ConvertToFormat(this System.Drawing.Image image, PixelFormat format)
         {
             Bitmap copy = new Bitmap(image.Width, image.Height, format);
             using (Graphics gr = Graphics.FromImage(copy))
             {
                 gr.DrawImage(image, new Rectangle(0, 0, copy.Width, copy.Height));
             }
             return copy;
         }
     }

If you press K, the mouse cursor will move to button1's place.

Note: It is necessary for you to Maximize Window otherwise the posistion is not correct.

Result:
187454-1.gif



Best Regards,
Jack


If the answer is the right solution, please click "Accept Answer" and upvote it.If you have extra questions about this answer, please click "Comment".

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.



image.png (16.8 KiB)
1.gif (15.2 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Valleze-6170 avatar image
0 Votes"
Valleze-6170 answered Valleze-6170 commented

I got this code from internet:


 Imports System.Drawing.Imaging
 Imports System.Runtime.InteropServices
    
 Public Class Form1
     Private ImgToFind As New Bitmap("C:\TestFolder\MyIconScreenShot.png")
    
     Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
         Dim rect As Rectangle = FindImageOnScreen(ImgToFind, False)
         If rect <> Rectangle.Empty Then
             Dim cntr As Point = New Point(rect.X + CInt(rect.Width / 2), rect.Y + CInt(rect.Height / 2))
             Cursor.Position = cntr
         Else
             MessageBox.Show("Image not found")
         End If
     End Sub
    
     ''' <summary>Finds a matching image on the screen.</summary>
     ''' <param name="bmpMatch">The image to find on the screen.</param>
     ''' <param name="ExactMatch">True finds an exact match (slowerer on large images). False finds a close match (faster on large images).</param>
     ''' <returns>Returns a Rectangle of the found image in sceen coordinates.</returns>
     Private Function FindImageOnScreen(ByVal bmpMatch As Bitmap, ByVal ExactMatch As Boolean) As Rectangle
         Dim ScreenBmp As New Bitmap(Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height)
         Using g As Graphics = Graphics.FromImage(ScreenBmp)
             g.CopyFromScreen(Screen.PrimaryScreen.Bounds.Location, Point.Empty, Screen.PrimaryScreen.Bounds.Size, CopyPixelOperation.SourceCopy)
         End Using
    
         Dim ImgBmd As BitmapData = bmpMatch.LockBits(New Rectangle(0, 0, bmpMatch.Width, bmpMatch.Height), ImageLockMode.ReadOnly, PixelFormat.Format24bppRgb)
         Dim ScreenBmd As BitmapData = ScreenBmp.LockBits(Screen.PrimaryScreen.Bounds, ImageLockMode.ReadOnly, PixelFormat.Format24bppRgb)
    
         Dim ImgByts((Math.Abs(ImgBmd.Stride) * bmpMatch.Height) - 1) As Byte
         Dim ScreenByts((Math.Abs(ScreenBmd.Stride) * ScreenBmp.Height) - 1) As Byte
    
         Marshal.Copy(ImgBmd.Scan0, ImgByts, 0, ImgByts.Length)
         Marshal.Copy(ScreenBmd.Scan0, ScreenByts, 0, ScreenByts.Length)
    
         Dim FoundMatch As Boolean = False
         Dim rct As Rectangle = Rectangle.Empty
         Dim sindx, iindx As Integer
         Dim spc, ipc As Integer
    
         Dim skpx As Integer = CInt((bmpMatch.Width - 1) / 10)
         If skpx < 1 Or ExactMatch Then skpx = 1
         Dim skpy As Integer = CInt((bmpMatch.Height - 1) / 10)
         If skpy < 1 Or ExactMatch Then skpy = 1
    
         For si As Integer = 0 To ScreenByts.Length - 1 Step 3
             FoundMatch = True
             For iy As Integer = 0 To ImgBmd.Height - 1 Step skpy
                 For ix As Integer = 0 To ImgBmd.Width - 1 Step skpx
                     sindx = (iy * ScreenBmd.Stride) + (ix * 3) + si
                     iindx = (iy * ImgBmd.Stride) + (ix * 3)
                     spc = Color.FromArgb(ScreenByts(sindx + 2), ScreenByts(sindx + 1), ScreenByts(sindx)).ToArgb
                     ipc = Color.FromArgb(ImgByts(iindx + 2), ImgByts(iindx + 1), ImgByts(iindx)).ToArgb
                     If spc <> ipc Then
                         FoundMatch = False
                         iy = ImgBmd.Height - 1
                         ix = ImgBmd.Width - 1
                     End If
                 Next
             Next
             If FoundMatch Then
                 Dim r As Double = si / (ScreenBmp.Width * 3)
                 Dim c As Double = ScreenBmp.Width * (r Mod 1)
                 If r Mod 1 >= 0.5 Then r -= 1
                 rct.X = CInt(c)
                 rct.Y = CInt(r)
                 rct.Width = bmpMatch.Width
                 rct.Height = bmpMatch.Height
                 Exit For
             End If
         Next
         bmpMatch.UnlockBits(ImgBmd)
         ScreenBmp.UnlockBits(ScreenBmd)
         ScreenBmp.Dispose()
         Return rct
     End Function
 End Class



I created a new form and just added a button to test this code above.

I added to the project:
AForge code library;
Aforge.Imaging library;
AForge.Math library;

There are many errors in this code, I couldn't fix it, but in this link below, we can see it working in the test that the author of the code published. Apparently, his code works without needing to maximize the form window.

Could you help me to fix the errors in this code, so that I can try to adapt it to my project? I'm using visual studio 2022, I don't know if this makes a difference to how the code works.

Here is the link where this code was posted, as an answer to another old question from someone else:

move-mouse-cursor-to-specific-icon-based-on-saved-image



· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@Valleze-6170, thanks for the respsponse, based on my test the above code can be complied sucessfully. However, it could not find the specific image. Please just remove the three nuget package, then you could complie the project.

1 Vote 1 ·

oh, I found my mistake. I confuse versions, but my project is in one version and this code above is in another. I'm not sure if the code above is in C# or visual basic. But my project is on another different version of this code, so it didn't work. Is it possible to adapt this code above to the other code version?

0 Votes 0 ·

@Valleze-6170, based on my test, the above code is not working for me to find the button correctly. Do you want to change this to c# version?

1 Vote 1 ·
Show more comments