Skip to content

Commit 513c484

Browse files
suarmagitsuarmagit
committed
initial public release
0 parents  commit 513c484

File tree

557 files changed

+28996
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

557 files changed

+28996
-0
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
*.pyc

‎LICENSE-murphy.txt

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
Copyright 2014 F-Secure Corporation.
2+
3+
Licensed under the Apache License, Version 2.0 (the "License");
4+
you may not use this file except in compliance with the License.
5+
You may obtain a copy of the License at
6+
7+
http://www.apache.org/licenses/LICENSE-2.0
8+
9+
Unless required by applicable law or agreed to in writing, software
10+
distributed under the License is distributed on an "AS IS" BASIS,
11+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
See the License for the specific language governing permissions and
13+
limitations under the License.

‎LICENSE-vncdotool.txt

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
Copyright (c) 2010 Marc Sibson
2+
3+
Permission is hereby granted, free of charge, to any person obtaining a copy
4+
of this software and associated documentation files (the "Software"), to deal
5+
in the Software without restriction, including without limitation the rights
6+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
7+
copies of the Software, and to permit persons to whom the Software is
8+
furnished to do so, subject to the following conditions:
9+
10+
The above copyright notice and this permission notice shall be included in
11+
all copies or substantial portions of the Software.
12+
13+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
14+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
15+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
16+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
17+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
18+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
19+
THE SOFTWARE.

‎LICENSE.txt

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
Murphy is licensed under the Apache 2.0 (APL 2.0) license, see
2+
LICENSE-murphy.txt
3+
4+
Murphy includes a slightly modified version of vncdotool in
5+
murphy\user_simulation\vnc\vncdotool:
6+
vncdotool is a command line VNC client. It can be useful to automating
7+
interactions with virtual machines or hardware devices that are otherwise
8+
difficult to control.
9+
See LICENSE-vncdotool.txt for license information, website can be found at
10+
https://github.com/sibson/vncdotool
11+

‎README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
murphy_sdk
2+
==========
3+
4+
Murphy open source

‎docs/extractingmodels.html

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
<html>
2+
<title>Your first model extraction scripts</title>
3+
<body>
4+
<h3>Starting from the samples</h3>
5+
<h4>Short explanation</h4>
6+
Create a dir 7zip_mine, copy extract_7zip_win_scraper.py inside of it.<br>
7+
In the line <pre>extractor = base_extractor.BaseExtractor('7zipWinScraper',</pre> change 7zipWinScraper for 7zip_mine<br>
8+
Add the new project to web_workbench/project.json files the following line:<br>
9+
<pre>,{"file": "../samples/7zip_mine/7zip_mine/7zip_mine.json"}</pre>
10+
Run extract_7zip_win_scraper.py<br>
11+
12+
<h4>Longer explanation</h4>
13+
It is ofthen easier to start with an existing script and adjust rather than totally from scratch.<br>
14+
For that purpose, create a folder called 7zip_mine (in the directory where you had murphy, for example: c:\git_projects\murphy_sdk\samples\7zip_mine<br>
15+
Copy the file extract_7zip_win_scraper.py from the samples\7zip into the new folder.<br>
16+
<br>
17+
The code of that file looks like this:<br>
18+
<pre>
19+
from model_extraction import base_extractor, configuration
20+
from model_extraction.ui import window_scrap
21+
22+
if __name__ == '__main__':
23+
test_files = configuration.get_default_config()["test files"]
24+
extractor = base_extractor.BaseExtractor('7zipWinScraper',
25+
'\\utils\\runurl.py ' + test_files + '7z920.exe')
26+
27+
#Ignore 'Browse For Folder' dialog as we dont care for windows native dialog
28+
extractor.add_boundary_node('Browse For Folder')
29+
30+
extractor.scrap_method = lambda node, world, scraper_hints, node_hints: window_scrap.custom_scraper(node, world)
31+
32+
extractor.crawl_application()
33+
</pre>
34+
We care about this line:<br>
35+
<pre>
36+
extractor = base_extractor.BaseExtractor('7zipWinScraper',
37+
'\\utils\\runurl.py ' + test_files + '7z920.exe')
38+
</pre>
39+
The first parameter is the name we give to the model, so lets start by changing it to 7zip_mine<br>
40+
The second parameter is the 1st action to perform when murphy starts crawling the application, we're not going to change it now but all you need to know for now is that
41+
it will be a command that when build will look like: <pre>\utils\runurl.py http://192.168.56.1:8901/files/7z920.exe</pre>
42+
In essence, it will download that url and then execute 7z920.exe<br>
43+
Once we modified that line it should look like:<br>
44+
<pre>
45+
extractor = base_extractor.BaseExtractor('7zip_mine',
46+
'\\utils\\runurl.py ' + test_files + '7z920.exe')
47+
</pre>
48+
We're ready to run it now, however we wont see it in the web workbench yet, we have to configure it, for doing so, open the file config.json that is located in the web_workbench directory.<br>
49+
It should look like this:<br>
50+
<pre>
51+
[
52+
{
53+
"file": "../samples/7zip/7zip/7zip.json"
54+
},
55+
{
56+
"file": "../samples/7zip/7zip_newer/7zip_newer.json"
57+
},
58+
{
59+
"file": "../samples/7zip/7zipWinScraper/7zipWinScraper.json"
60+
}
61+
]
62+
</pre>
63+
It is pretty self described, we need to copy the last entry and add our new model, you should modify it as:<br>
64+
<pre>
65+
[
66+
{
67+
"file": "../samples/7zip/7zip/7zip.json"
68+
},
69+
{
70+
"file": "../samples/7zip/7zip_newer/7zip_newer.json"
71+
},
72+
{
73+
"file": "../samples/7zip/7zipWinScraper/7zipWinScraper.json"
74+
},
75+
{
76+
"file": "../samples/7zip_mine/7zip_mine.json"
77+
}
78+
]
79+
</pre>
80+
If you dont have the workbench running, launch it now (for example: C:\git-projects\murphy_sdk\web_workbench>workbench.py)<br>
81+
Open a browser to http://127.0.0.1:8090 and enter your name, the project should appear in the dropdown list of models, of course it will still display a blank model.<br>
82+
(if you had a browser opened then reload the page so you'll get the new project)<br>
83+
So from a command line do:<br>
84+
<pre>C:\git-projects\murphy_sdk\samples\7zip_mine>extract_7zip_win_scraper.py</pre>
85+
In a few seconds you should see:
86+
<pre>
87+
2014-01-27 15:20:29,338 - root.model_extraction.crawler - INFO - Started exploration, current node is None, current edge is None
88+
2014-01-27 15:20:29,556 - root.model_extraction.crawler - INFO - Node created for state Node Node 0 (1)
89+
->Launch application->None
90+
2014-01-27 15:20:29,571 - root.model_extraction.ui.edge - INFO - Performing Edge Launch application (Node 0)
91+
2014-01-27 15:20:29,782 - root.murphy.user_simulation.helpers - INFO - Waiting for a stable screen
92+
...
93+
</pre>
94+
Go now to the web workbench page and select the model from the dropdown list, you may have to wait a little until first images are viewable.<br>
95+
<br>
96+
<h4>If something went wrong</h4>
97+
Make sure you had PYTHONPATH set (for example set PYTHONPATH=C:\git-projects\murphy_sdk).<br>
98+
Ensure that when you are going to execute the script (extract_7zip_win_scraper.py) the virtual machine IS NOT RUNNING in virtualbox.<br>
99+
</body>
100+
</html>

‎docs/how.html

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
<html>
2+
<title>How murphy works</title>
3+
<body>
4+
<h3>High level inner workings of murphy</h3>
5+
<h4>Extracting an application model</h4>
6+
Murphy will run in one machine (let's call it the controller) and will spawn a virtual machine<br>
7+
In the virtual machine it will launch the application to be analyzed<br>
8+
It will analyze what is seeing in the screen, what is the currently active window, what actions can the user do (buttons / links it can click, text inputs, selections from dropdown boxes, etc)<br>
9+
It will systematically try to do those actions and generate a model that describes such application<br>
10+
At any given point, murphy may decide to recreate the virtual machine to get back to a clean state (to prevent side effects from previously executed actions)<br>
11+
<h4>Separation of concerns</h4>
12+
Internally murphy has a few abstractions, amongst them:<br>
13+
<ul>
14+
<li>A scraper (a sorry name, but responsible to decompose what the user sees, akin to a DOM of the application)</li>
15+
<li>A crawler (responsible for 'walking around' the application for mapping it)</li>
16+
<li>A machine (handles the logic for creating, allocating and deallocating a virtual machine or device</li>
17+
<li>A user (simulates user actions like mouse clicks and keyboard strokes, also responsible for providing screenshots</li>
18+
</ul>
19+
<h4>How does murphy know what elements the user can interact with?</h4>
20+
The scraper component is a class responsible for doing that, there are at the moment 2 scrapers provided in murphy.<br>
21+
One uses windows API's for enumerating windows and controls, so any native application built on c, c++, mfc and many other technologies will work fine,
22+
it can be found in model_extraction_helpers/window_scraper.py<br>
23+
The other uses user actions (pressing the TAB key and hovering the mouse) and analyzes the changes in the UI, it then deduces the user interface elements based on that, it can be found in model_extraction/ui/scraper.py<br>
24+
The good side of scraper.py is that it works for most applications independently of the language used to develop them, the bad is that is unable to extract texts from the ui and that many applications are poorly written
25+
and many ui elements are not reachable while pressing the TAB key.<br>
26+
<br>
27+
It is relatively simple to create additional scrapers, in house we had to develop a custom one for applications based on QT 4, we also develop an experimental scraper for android applications and also another one
28+
based on the windows accessibility API (ui automation)<br>
29+
<h4>How does murphy know it has to input a serial key / activation code / etc in an input field?</h4>
30+
It doesnt, that's why the way to extract a model is to write a simple script and tell murphy the very specific, non guessable things of the application you're interested in.<br>
31+
<h4>Boundaries</h4>
32+
You dont want to crawl everything, suppose a case that there's a link in your application that opens a browser, im pretty sure you dont want murphy to crawl the whole internet.<br>
33+
Here's where the concept of boundary comes, in an extraction script, one of the things you tell murphy is what you're NOT interested to crawl, in other words what are the boundaries of the exploration<br>
34+
<h4>In simple words</h4>
35+
The idea is to tell only the essential things in an extraction script, some typical things to handle:<br>
36+
<ul>
37+
<li>launch x application (run x command)</li>
38+
<li>if the window is a help window, i'm not interested to crawl beyond it.</li>
39+
<li>if the window is the 'enter serial number' then use x value on y field</li>
40+
<li>if the window is 'license information' then the value of field 'y' must be ignored or custom handle (for example a timestamp)</li>
41+
</ul>
42+
<h4>Simplest model extraction script</h4>
43+
<pre>
44+
extractor = base_extractor.BaseExtractor('7zip', '\\utils\\runurl.py http://127.0.0.1:8901/files/7z920.exe')
45+
46+
extractor.add_boundary_node('Browse For Folder')
47+
48+
extractor.crawl_application()
49+
</pre>
50+
</body>
51+
</html>

‎docs/img/7zip-desc.png

16.9 KB
Loading

‎docs/img/7zip-flow.png

61.9 KB
Loading

‎docs/img/diff.png

46.8 KB
Loading

‎docs/img/diff2.png

69.8 KB
Loading

‎docs/img/export.png

21.8 KB
Loading

‎docs/img/inputs.png

31.6 KB
Loading

‎docs/img/new_model.png

88.5 KB
Loading

‎docs/img/old_model.png

69.2 KB
Loading

‎docs/img/planning.png

95.5 KB
Loading

‎docs/img/realflow.png

88.8 KB
Loading

‎docs/img/route.png

41.3 KB
Loading

‎docs/img/runlog.png

47.8 KB
Loading

‎docs/img/smallflow.png

37 KB
Loading

‎docs/install.html

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
<html>
2+
<title>Installing murphy</title>
3+
<body>
4+
<br>
5+
<strong>WARNING</strong><br>
6+
Murphy will launch a web server in the machine it is installed and will listen for connections in port 8090, care has been taken as this to be as safe as possible, however it
7+
is recommended that you disable inbound connections to that port in your firewall unless you know and are sure of what you're doing.<br>
8+
<h3>Platforms</h3>
9+
Windows 7 (should work in Vista and windows 8, maybe xp, never tested with Metro apps)<br>
10+
There's interest in making it work under Linux and we're almost there, if you know linux you can probably make it work.<br>
11+
<h3>Dependencies</h3>
12+
Python 2.7.3 or newer (32 bits), wont probably work on python 3.x and I havent tested with pre 2.7.3. It can be found <a href="http://www.python.org/download/releases/2.7/">here</a><br>
13+
There's a small dll written in c and compiled for 32 bits platform so it wont work on python 64 bits, however the work needed to support it is minimal<br>
14+
Python PIL (be sure to install the one that matches your python version, including 32 or 64 bits...), it can be found <a href="http://www.pythonware.com/products/pil/">here</a><br>
15+
The included vnc library uses Twisted (download twisted for python), which can be found <a href="https://twistedmatrix.com">here</a><br>
16+
Twisted needs zope (4.0.6 seems to work fine), and can be found <a href="https://pypi.python.org/pypi/zope.interface">here</a><br>
17+
Graphviz, which is used for the flow graphs can be found <a href="http://www.graphviz.org/">here</a> (Note: latest versions has some issues with the generated SVG, version 2.32 is known to work fine)<br>
18+
You'll need Bottle, you can get it from <a href="http://bottlepy.org/">here</a>, just drop the file in the web_workbench directory (it is known to work with version 0.11)<br>
19+
VirtualBox, it can be found <a href="https://www.virtualbox.org/">here</a><br>
20+
Other virtualization tools like Kvm, VmWare, Dvmps, etc. can be used but need a small and simple adapter, take a look at virtualbox.py and rules_vbox.py in the model_extraction directory for that matter.<br>
21+
7zip is used in some examples / exercises, get 7z920.exe and put it in the directory model_extraction_helpers/files (newer versions may work but I haven't tested them)<br>
22+
23+
24+
<h3>Murphy</h3>
25+
Since you're reading this, you already have this or know how to get this.<br>
26+
You can copy murphy to any directory, you'll need to set your PYTHONPATH to that directory, so if you did a git clone to say c:\git_projects\murphy_sdk then do set PYTHONPATH=c:\git_projects\murphy_sdk<br>
27+
Note that PYTHONPATH is needed for the model extraction scripts, the web workbench doesn't need it.<br>
28+
29+
<h3>Virtual machines</h3>
30+
Murphy uses virtual machines to run the applications it will crawl and extract their models, it will not run the applications locally (unless you modify murphy to do so)<br>
31+
See <a href="virtual machine.html">here</a> on instructions on how to create a virtual machine that murphy can use.<br>
32+
<h3>Checking the installation</h3>
33+
Last but not least, in the murphy folder, there's a file graphviz.py, change the line:
34+
<pre>
35+
_GRAPHVIZ_EXE = (r'\Program Files (x86)\Graphviz2.32\bin\dot.exe')
36+
</pre>
37+
As to point to the version of graphviz you installed<br>
38+
After taking care of the dependencies, creating and configuring a virtual machine, be sure the virtual machine image you're going to use is not running!<br>
39+
<br>
40+
Assuming that your installation directory is c:\git_projects\murphy_sdk, do:<br>
41+
<pre>
42+
c:\git_projects\murphy_sdk\web_workbench>python workbench.py
43+
Workbench running on pid 736
44+
Bottle v0.11.rc1 server starting up (using WSGIRefServer())...
45+
Listening on http://0.0.0.0:8090/
46+
Hit Ctrl-C to quit.
47+
</pre>
48+
Open the browser of your choice, I use chrome and every now and then test that it works in FF, no clue if it even works in IE.<br>
49+
Point your browser to http://127.0.0.1:8090/<br>
50+
When asked about your user name you can put whatever you want as it does not do any authentication, press the login button.<br>
51+
At the top-left corner you should see a dropdown with the models, choose one from the list.<br>
52+
After a few seconds you should see a statechart loaded in the browser, if not take a look a the console output as something went bad.<br>
53+
Launch the webserver that provides the test files:<br>
54+
<pre>C:\git-projects\murphy_sdk\model_extraction_helpers>python -m SimpleHTTPServer 8901</pre>
55+
<h4>Last but not least, test that the model extraction works</h4>
56+
From a console, do (remember to set the PYTHONPATH var!)<br>
57+
<pre>
58+
C:\git-projects\murphy_sdk\samples\7zip>extract_7zip_win_scraper.py
59+
</pre>
60+
After a few seconds you should see something similar to this:<br>
61+
<pre>
62+
C:\git-projects\murphy_sdk\samples\7zip>extract_7zip_win_scraper.py
63+
2014-01-26 12:59:07,364 - root.model_extraction.crawler - INFO - Started exploration, current node is None, current edge is None
64+
2014-01-26 12:59:07,641 - root.model_extraction.crawler - INFO - Node created for state Node Node 0 (1)
65+
->Launch application->None
66+
2014-01-26 12:59:07,654 - root.model_extraction.ui.edge - INFO - Performing Edge Launch application (Node 0)
67+
2014-01-26 12:59:07,897 - root.murphy.user_simulation.helpers - INFO - Waiting for a stable screen
68+
2014-01-26 12:59:15,072 - root.murphy.user_simulation.vnc.vncdotool.vnc_wrapper - INFO - Sending keystrokes {+ctrl}{esc}{-ctrl}
69+
2014-01-26 12:59:15,842 - root.murphy.user_simulation.vnc.vncdotool.vnc_wrapper - INFO - Sending keystrokes \utils\runurl.py http://192.168.56.1:8901/files/7z920.exe{enter}
70+
2014-01-26 12:59:31,237 - root.murphy.user_simulation.vnc.vncdotool.vnc_wrapper - INFO - moving mouse to 1 1
71+
2014-01-26 12:59:31,453 - root.model_extraction.virtualbox - INFO - Wait for remote machine to be idle...
72+
2014-01-26 12:59:31,456 - root.murphy.user_simulation.helpers - INFO - Waiting for a stable screen
73+
2014-01-26 12:59:39,292 - root.model_extraction.ui.node - DEBUG - Checking if i'm in Node 0
74+
...
75+
</pre>
76+
It will take something between 5 and 10 minutes to extract the model, depending on many many many factors but it should finish, if you notice
77+
in the log output that got stuck for more than a couple of minutes then something is definitively wrong.<br>
78+
After (or while) the model is extracted, you can check the progress in the workbench tool.<br>
79+
The models are already extracted in the git version so you can see the flows in the workbench but they wont work interactively or thru scripting if you dont rebuild them,
80+
now is a good time to build the other provided examples (or you're doomed to forget it :))<br>
81+
</body>
82+
</html>

‎docs/introduction.html

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
<html>
2+
<title>Murphy</title>
3+
<body>
4+
<h1>Murphy, a set of tools for automation.</h1>
5+
<br>
6+
<img src="img/smallflow.png"><br>
7+
<br>
8+
Some of the things that can be done with can be found <a href="overview.html">here</a>.<br>
9+
<br>
10+
<a href="why.html">Why?</a><br>
11+
<br>
12+
<a href="how.html">How?</a><br>
13+
<br>
14+
<a href="worthknowing.html">Things you should know...</a><br>
15+
<br>
16+
<a href="install.html">Installation & running the samples</a><br>
17+
<br>
18+
<a href="extractingmodels.html">Creating your first model extraction scripts</a><br>
19+
<br>
20+
<a href="quickref.html">Quick references</a><br>
21+
<br>
22+
More to come...<br>
23+
<br>
24+
<h3>References on the web</h3>
25+
A paper related to murphy was presented in EESSMod 2013 - Experiences and Empirical Studies in Software Modeling at <a href="http://ceur-ws.org/Vol-1078/">http://ceur-ws.org/Vol-1078/</a>, a direct
26+
link to the paper is <a href="http://ceur-ws.org/Vol-1078/paper6.pdf">here</a><br>
27+
Internal workshop of Ericsson in Stockholm, 2013 (sorry, no online material available at the moment)<br>
28+
Anoter presentation related to murphy was submitted to <a href="http://issta2014.org/">ISSTA 2014</a><br>
29+
<br>
30+
There's no official support, we may have support forums in the future if there's enough interest, in the meaintime (and I make no promises) you can try reach me at valkoinen.rapu at gmail.com.<br>
31+
Some basic info I've written can be found in <a href="http://valkoinenrapu.blogspot.com/2014/01/about-time.html">this blog</a><br>
32+
<br>
33+
<h3>Thanks to</h3>
34+
Everybody who made this possible including the vncdotool guys and Pekka Aho et al (who kindly published papers about the experiences of Murphy in F-Secure)<br>
35+
</body>
36+
</html>

0 commit comments

Comments
 (0)