IExplorer="Image=C:\SAFSProject\Datapool\IExplorer\AnchorIcoVista.BMP"
Relative Path:
IExplorer="Image=IExplorer\"
Contents: Walk-Thru, Syntax, Images, Commands, more...
One of the big issues facing todays test automator is the fast-pace in which tools and application technologies are changing. Applications might be developed in Java Swing, or RCP\SWT, or .NET 2.0, or 3.0, AJAX, Adobe Flex, Google's GWT\Chrome, Delphi, or PowerBuilder; and they might be running in IE6, or IE7, or Firefox 2.0, or Firefox 3.0 or who-knows-what else.
Tool manufacturers can hardly keep pace with the needs of cutting edge application testers. If you are tasked with testing newer or novel technologies then it is usually difficult to find a tool that can work in that environment early in the development and testing lifecycle. When the tools you have can't support the technologies you need to test you need to turn to something else. This is a good time to consider Image-Based testing.
Image-Based testing allows us to test virtually anything that can be displayed on the screen. It doesn't matter what the underlying technology is, if it is visible on the screen then we can interact with it.
The SAFS Framework to-date has largely been based on "Object-Based" testing and SAFS Component Recognition. With Image-Based testing we must now expand our recognition syntax to allow for testing based on finding and interacting with graphics on the screen.
Test records for image-based testing are the same as they are for object-based testing:
T, IExplorer, Maximize, ClickTest Records:
T, IExplorer, Restore, Click
T, IExplorer, Close, Click
The friendly names for the "window" and the "component" are mapped in the application map the same as for object-based tests, but the syntax facilitates image-based testing:
[IExplorer]IExplorer App Map Entry:
IExplorer="Image=<imagepath>"
<imagepath can be the full path to a single image or to a directory containing multiple images. Multiple images are necessary if the target image is different in different environments. For example, on different platforms, or different versions of the application or operating system. The framework will search the screen for each of the images in the directory until it finds the match.
<imagepath> can also be an incomplete 'relative' path to a single image or to a directory of images. The specified path can be relative to the SAFS project directory, or to the SAFS project\Datapool directory.
Sample <imagepath>:
Right away we can tell that finding the IExplorer icon in the top-left of the titlebar does not give us enough information to locate significant areas within the IExplorer window. We can find any individual image this way and act on areas relative to it. But in many cases it will be beneficial to identify more specific bounds for the area we want to search or act upon.
We can enhance our IExplorer recognition to include an image to the right of the anchor that identifies the width or right-side bounds of the area of interest. We can add an ImageRight or ImageR modifier to find the Close icon in the titlebar. The framework knows to search only within the narrow band to the right of the anchor icon for this ImageR image.
Enhanced <imagepath>:
By capturing these two small images, we are now able to complete our App Map and execute all three of the test records listed above. This works because we used two images to define the area of interest of our "window" and the framework knows to only seek "components" within this area of interest.
[IExplorer]IExplorer App Map Entries:
IExplorer="Image=IExplorer\;ImageR=Titlebar\Close\"
Maximize="Image=Titlebar\Close\;Hotspot=-17"
Restore="Image=Titlebar\Close\;Hotspot=-17"
Minimize="Image=Titlebar\Close\;Hotspot=-35"
Close="Image=Titlebar\Close\"
As you see we can also define Hotspots that tell the framework to act on a point relative or offset from the location where an image was actually found. We specified Hotspot=-12 (12 pixels to the left) of the Close icon to act on the Maximize and Restore controls without having to capture their images.
Sample:
IExplorer="Image=<imagepath>"
Sample:
IExplorer="Image=<imagepath>;ImageR=<imagepath>"
Samples:
IExplorer="Image=<imagepath>;ImageB=<imagepath>"
IExplorer="Image=<imagepath>;ImageR=<imagepath>;ImageB=<imagepath>"
IExplorer="Image=<imagepath>;ImageB=<imagepath>;ImageR=<imagepath>"
Samples:
IExplorer="Image=<imagepath>;Index=3"
IExplorer="Image=<imagepath>;Index=3;ImageR=<imagepath>"
Samples:
IExplorer="Image=<imagepath>;BitTolerance=70"
IExplorer="Image=<imagepath>;ImageR=<imagepath>;BT=75"
Samples:
Close="Image=<imagepath>;Hotspot=offsetX,offsetY"
Close="Image=<imagepath>;Hotspot=-12"
Close="Image=<imagepath>;Hotspot=-12,0"
Close="Image=<imagepath>;Hotspot=-12 3"
Close="Image=<imagepath>;Hotspot= ,3
Close="Image=<imagepath>;Hotspot=0,3
Valid PR= values are:
TopLeft TL TopCenter TC TopRight TR LeftCenter LC Center C RightCenter RC BottomLeft BL BottomCenter BC BottomRight BR
Samples:
Maximize="Image=<imagepath>;PointRelative=LeftCenter"
Maximize="Image=<imagepath>;PR=LC;Hotspot=-5" (5 pixels from the left edge vertically centered)
Maximize="Image=<imagepath>;HS=0,5;PointRelative=BottomRight" (5 pixels below the bottom right corner)
For component searches, SearchRect can now be used to expand or modify the area to be searched--not just reduce the scope of a search. This enables very creative component searches that can actually be almost anywhere relative to the window rectangle previously found.
X,Y,W,and H are individually optional and their presence or absence will define or modify the search rectangle accordingly. Missing X and\or Y values will default to 0. Missing W and\or H values will default to the maximum width and height of the screen when searching for a window image and default to the width and height of the window rectangle when searching for a component relative to the window.
Coordinates can be comma OR space delimited, but only use one or the other.
All values can be absolute or can be specified as a percentage. For component searches relative to a window the percentage is not limited to 100%. For example, it is reasonable to expand the width of a search rectangle by more than 100%. All percentages for component searches are considered to be percentage of window width and height NOT window position(X,Y) or screen width and height(W,H).
Window Search Samples:
TopEdgeItem="Image=<imagepath>; SearchRect=0,0, ,75" (Search only the top 75 pixels) RightEdgeItem="Image=<imagepath>;SR=750,0" (Start search 750 pixels from left) LeftEdgeItem="Image=<imagepath>; sr=0,0,25%" (Search only the left 25% of screen) TaskBarItem="Image=<imagepath>; SR=0,90%" (Search only the bottom 10% of screen)
Component Search Samples:
a) TitleBarItem="Image=<imagepath>; SearchRect= , , , 10%" b) OffsetTitlebarItem="Image=<imagepath>;SR=0,-15, 10, 120%" c) LeftSideItem="Image=<imagepath>; sr= -150%, -5, 160%, 30"
a) The TitleBarItem SearchRect above indicates that only the top 10% of the found window rectangle should be searched for the required component image. This limits the search to what is often considered to be the Titlebar of a window.
(Of course, we aren't always looking for a window. Sometimes we are just looking for a reference image anywhere on the screen.)
EXAMPLE TitleBarItem SearchRect= 0, 0, 0, 10%
Modified rectangle for the component search is: x=400, y=200, w=100, h=30
b) The OffsetTitlebarItem SearchRect shows we want to start our search 15 pixels higher (y-15) than we otherwise would search. This is useful if the component image we are seeking is not exactly inline or inside the window rectangle we are working with. This SearchRect is also expanding the width of our search by 10 pixels (w+10) and the height of our search by 120% (h*120%).
(We often want to alter the width and\or height of the search to accomodate changes we might have made to the x and y coordinates for the start of the search.)
EXAMPLE OffsetTitlebarItem SearchRect= 0, -15, 10, 120%
Modified rectangle for the component search is: x=400, y=185, w=110, h=360
c) The LeftSideItem SearchRect shows an example where we aren't actually looking for something inside our "window". Here we are actually trying to find an image that is actually to the left of our initial window image--outside the bounds of the "window". In this case, we are changing the component search rectangle 'x' coordinate to move left of the window by 150% (w*150%). If the window rectangle is 50 pixels wide then we are moving the start of our component search(x) 75 pixels to the left of the window rectangle. This SearchRect is also expanding our search rect 5 pixels higher(y-5), making the search width 160% greater(w*160%), and adding 30 pixels to the height of the search(h+30).
EXAMPLE LeftSideItem SearchRect= -150%, -5, 160%, 30
Modified rectangle for the component search is: x=250, y=195, w=160, h=330
Notice how the LeftSideItem will be sought to the left of the original "window" and not inside it.
Icons or images stored for screen matching must be in one of the following formats:
More image formats will be supported in the future.
It is also important to note there are differences in display configurations that will likely require separate images to support that display. For example, performing the same test in the following display configurations will likely require a different set of images for each configuration:
This is not an issue of screen resolution. Images stored for a particular Display typically work for most or all screen resolutions on that Display.
This is an issue that each Display is configured for different levels of data compression. Bitmaps stored for the Normal Display have no data compression and no loss of image information. The displayed image for the Remote displays is compressed and there is loss of image information. Because of this, Normal Display images will not match Remote Display images.
This is something we hope to work on in the near future.
Note:*** Remote testing over VNC does NOT have this display problem! *** The target machine is actually using its Normal Display (uncompressed) so the images that work for the Normal Display continue to work locally even when manipulated or viewed remotely.
The following commands are supported for Image-Based Testing:
| Click | a.k.a. ClickClickScreenImage |
| RightClick | a.k.a. RightClickScreenImage |
| DoubleClick | a.k.a. DoubleClickScreenImage |
| LeftDrag | |
| RightDrag | |
| InputKeys | a.k.a. TypeKeys |
| InputCharacters | a.k.a. TypeChars |
| GetGUIImage | |
| VerifyGUIImageToFile | |
| GuiDoesExist | |
| GuiDoesNotExist |
T, WindowID, WindowID , Click
T, WindowID, CompID , Click, "Coords=20;45"
T, Notepad , CloseIco , RightClick
T, Notepad , Titlebar , DoubleClick
T, WindowID, CompID , LeftDrag, Left2Right
T, WindowID, CompID , LeftDrag, "10;10;200;20"
T, Notepad , Notepad , InputCharacters, "Any Text Here"
T, Notepad , Notepad , InputKeys, "{TAB}{DOWN 3}{ENTER}%{F4}"
T, Notepad , Notepad , InputKeys, "%{F4}"
The Click commands do not yet support the AppMapSubKey parameter as documented in the SAFS Keyword Reference.
Consult the InputKeys Map for the format of keystrokes for InputKeys.
Driver Commands:
C, WaitForGui , WindowID , WindowID, 10
C, WaitForGui , WindowID , CompID , 15
C, WaitForGuiGone , WindowID , WindowID, 30
C, WaitForGuiGone , WindowID , CompID , 5
The WaitFor commands do support the default 15 second timeout when TIMEOUT is not specified.
WindowID="Image=[pathTo]\image.ext;[hotspot=x[,y][;pointrelative=constant]" =================================================================================This sample syntax shows a single image defining a component or window. It shows the optional use of specifying a hotspot and an optional pointrelative location for that hotspot.
If image fullpath is not provided a path relative to the Project is assumed. If project-relative path is not found then path relative to project\Datapool is assumed.
For a WindowID, the bounds of the single image specifies the total bounds of the deduced rectangle for that "Window" object. A subsequent CompID search will first attempt to be found within the bounds of that "Window". If not found, then the top-left corner of the "Window" rectangle defines the top-left coordinate of the remaining area to search on the screen for the CompID image.
Example:
1. WindowID="Image=\image.ext;HotSpot=2,2;PointRelative=TopLeft" (hotspot is 2,2 pixels in from the top-left corner of the deduced rectangle) 2. WindowID="Image=\image.ext;hs=2,2;pr=tl" (hotspot is 2,2 pixels in from the top-left corner of the deduced rectangle) 3. WindowID="Image=\image.ext;HS=-10;pr=LeftCenter" (hotspot is 10 pixels to the left and vertically centered on the left edge of the deduced rectangle) 4. WindowID="Image=\image.ext;hotspot= ,-10;pr=TopCenter" (hotspot is 10 pixels above and horizontally centered on the top edge of the deduced rectangle) 5. WindowID="Image=\image.ext;pr=BottomRight" (hotspot is the bottom-right corner of the deduced rectangle) WindowID="Image=[pathTo]\image1.ext;ImageRight=[pathTo]\image2.ext;....." ===============================================================================This sample syntax shows a double image defining for a component or window. These two images are intended to define the top-left and top-right corners of the object's deduced rectangle (once both images are found).
If image fullpath is not provided a path relative to the Project is assumed. If project-relative path is not found then path relative to project\Datapool is assumed.
The top-left image1 will be sought first. Once found,the top-right image2 will be sought within the vertically limited bounds defined by top-left image1 along with any Top and Bottom OutSets, if provided.
For a WindowID, coordinates for top-left image1 and top-right image2 deduces the top-left and width of the bounds for the deduced rectangle for that "Window" object. The height of the window will be from that deduced top edge down to the bottom of the screen. A subsequent CompID search will be limited within these bounds of that "Window".
Example:
1. WindowID="Image=\topleft.ext;ImageR=\topright.ext;HotSpot=0,2;PointRelative=TopCenter" Hotspot is 2 pixels down from the top-center edge of the deduced rectangle. The width of the deduced rectangle is limited by the outer coordinates at which topright.ext was found. 2. WindowID="Image=\topleft.ext;ImageR=topright.ext;hs=-10;pr=LeftCenter> Hotspot is 10 pixels to the left and vertically centered on the left edge of the deduced rectangle. The width of the deduced rectangle is limited by the outer coordinates at which topright.ext was found.
Recall that ImageBottom(ImageB) can be used to similarly limit the deduced height of the rectangle. ImageB can be used with the anchor Image alone, or in conjunction with ImageRight to fully define the width and height of the target window or component.
Top, Walk-Thru, Syntax, Images, Commands, more...