Turn Whiteboard UX Sketches into Working HTML in Seconds - Introducing Sketch2Code

  • Article

This post is authored by Tara Shankar Jana, Senior Technical Product Marketing Manager at Microsoft.

The user interface design process involves lots of creativity and iteration. The process often starts with drawings on a whiteboard or a blank sheet of paper, with designers and engineers sharing ideas and trying their best to represent the underlying customer scenario or workflow. Once a candidate design is arrived at, it's usually captured via a photograph and then translated manually into a working HTML wireframe that works in a web browser. Such translation takes time and effort and it often slows down the design process.

What if the design could instead be captured from a whiteboard and be instantly reflected in a browser? If we could do that, at the end of a design brainstorm session we would have a readymade prototype that's already been validated by the designer, developer and perhaps even the customer.

Introducing Sketch2Code – a web based solution that uses AI to transform a picture of a hand-drawn user interface into working HTML code.

Let's take a closer look at the process of transforming hand-drawn images into HTML using Sketch2Code:

  • The user first uploads an image using our website.
  • A custom vision model predicts what HTML elements are present in the image and also pins their location.
  • A handwriting text recognition service reads the text inside the predicted elements.
  • A layout algorithm uses the spatial information from the bounding boxes of the predicted elements to generate a grid structure that accommodates all these components.
  • An HTML generation engine uses the above pieces of information to generate HTML markup code reflecting the end result.

The application workflow looks something like this:

Sketch2Code uses the following elements:

  • Microsoft Custom Vision Model: This model has been trained with images of different handwritten designs, tagging the information associated with common HTML elements including text boxes, buttons, images, etc.
  • Microsoft Computer Vision Service: This is used to identify the text within a design element.
  • Azure Blob Storage: The information associated with each step of the HTML generation process is stored, including the original image, predicted results, the layout and grouping information, etc.
  • Azure Function: This serves as the backend entry point that coordinates the generation process by interacting with all the services.
  • Azure Website: The user interface front-end that enables uploading a new design and seeing the generated HTML results.

The above elements combine together via the following architecture:

Intrigued? You can find the code, solution development process and all other details associated with Sketch2Code on GitHub. Sketch2Code was developed by Microsoft in collaboration with Kabel and Spike Techniques.

We find the range of scenarios in which AI can be applied to be truly amazing, and this is one simple but powerful example of how AI can augment human ingenuity. If you get a chance to play with Sketch2Code, please do share your experiences and thoughts with us below.

Tara