Making the w3af module
From OWASP Live CD 2008
Contents |
Get the Source for w3af
While you can go to the w3af website and which has a link to the sourceforge site, I didn't do that. Instead, I used subversion to pull the latest version directly from the project's Subversion trunk onto the Live CD. Not only will you get the latest, greatest w3af has to offer, but you'll also be able to easily update the software again. (3 cheers for the svn client!)
Go ahead and pull down the trunk into the /opt/owasp directory
# cd /opt/owasp # svn co https://w3af.svn.sourceforge.net/svnroot/w3af/trunk w3af # cd w3af # svn info Path: . URL: https://w3af.svn.sourceforge.net/svnroot/w3af/trunk Repository Root: https://w3af.svn.sourceforge.net/svnroot/w3af Repository UUID: 16c29cf1-982c-0410-8ff8-8bb040e68b5b Revision: 1903 Node Kind: directory Schedule: normal Last Changed Author: andresriancho Last Changed Rev: 1902 Last Changed Date: 2008-10-26 16:11:29 +0000 (Sun, 26 Oct 2008)
NOTE: By the time I created the actual module, the release number was much higher - 2161. The svn release number is in the module name. Don't forget to use the svn client module to update to the latest from svn in the running Live CD.
Install any prerequisites for w3af
You will need to make sure that the SQLite module is installed. You can check for it by following the instructions Add modules to a running system and just looking at what comes up in the initial list of installed modules. If necessary, go here and grab it. The same is true for the Graphviz module which is also available on the Google Code site.
Props to the unidentified individual at http://www.nukeit.org/ who was nice enough to share his/her script to get w3af working on Backtrack 3 by making a post to the Live CD 2008 list. They've got the script posted here. I used it as a guide for my second attempt at getting w3af working. You can read about my first failed attempt at w3af module fail.
Based on his script, he lists the following dependencies to add to Backtrack 3:
- Python 2.5.2 - Already provided on the Live CD as a module see Supporting Software List
- glib2-2.14.6 - Already provided on the Live CD in the SLAX core module
- pango-1.20.0 - Already provided on the Live CD in the SLAX xorg module
- gtk+2-2.12.9 - Already provided on the Live CD in the SLAX xorg module
- pygobject-2.14.1 - Not present. Will add to the w3af module
- pygtk-2.12.1 - Not present. Will add to the w3af module
- pysqlite-2.4.1 - Not present. I'm going to build the latest from source after exchanging some emails with Andres Riancho (w3af project lead)
- pydot-1.0.2 - No longer a dependency per email to the w3af-users list titled "Removing dependencies and adding more performance" on Sun, Oct 26, 2008 at 11:18 AM by Andres Riancho. The svn checkout is after his commit.
- pyparsing-1.5.0 - Same as pydot above.
- SOAPpy, cluster, fpconst, pyPDF, pygoogle & pywordnet which all part of the svn checkout.
For those that have Slackware packages, I'll create .lzm's directly from the .tgz packages. First, I'll download them and verify the signatures:
$ wget http://slackware.mirrors.tds.net/pub/slackware/slackware-12.1/slackware/l/pygobject-2.14.1-i486-1.tgz $ wget http://slackware.mirrors.tds.net/pub/slackware/slackware-12.1/slackware/l/pygobject-2.14.1-i486-1.tgz.asc $ gpg --verify pygobject-2.14.1-i486-1.tgz.asc pygobject-2.14.1-i486-1.tgz gpg: Signature made Tue 08 Jan 2008 10:53:38 PM CST using DSA key ID 40102233 gpg: Can't check signature: public key not found $ gpg --keyserver wwwkeys.pgp.net --recv-keys 40102233 gpg: requesting key 40102233 from hkp server wwwkeys.pgp.net gpg: key 40102233: public key "Slackware Linux Project <security@slackware.com>" imported gpg: no ultimately trusted keys found gpg: Total number processed: 1 gpg: imported: 1 $ gpg --verify pygobject-2.14.1-i486-1.tgz.asc pygobject-2.14.1-i486-1.tgz gpg: Signature made Tue 08 Jan 2008 10:53:38 PM CST using DSA key ID 40102233 gpg: Good signature from "Slackware Linux Project <security@slackware.com>" gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: EC56 49DA 401E 22AB FA67 36EF 6A44 63C0 4010 2233 $ wget http://slackware.mirrors.tds.net/pub/slackware/slackware-12.1/slackware/l/pygtk-2.12.1-i486-2.tgz $ wget http://slackware.mirrors.tds.net/pub/slackware/slackware-12.1/slackware/l/pygtk-2.12.1-i486-2.tgz.asc $ gpg --verify pygtk-2.12.1-i486-2.tgz.asc pygtk-2.12.1-i486-2.tgz gpg: Signature made Wed 19 Mar 2008 07:41:00 PM CDT using DSA key ID 40102233 gpg: Good signature from "Slackware Linux Project <security@slackware.com>" gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: EC56 49DA 401E 22AB FA67 36EF 6A44 63C0 4010 2233
Now that I have some known-good sources and I've moved them over to the Live CD, I'll create modules out of them and install them via the command line installer (activate). I copy the .lzm's out of the working directory because activate removes the module's file after it installs it.
# mkdir working
# cd working
[scp them over or use method of your choice]
# tgz2lzm pygobject-2.14.1-i486-1.tgz pygobject-2.14.1.lzm
# tgz2lzm pygtk-2.12.1-i486-2.tgz pygtk-2.12.1.lzm
# cp *.lzm ../
# cd ../
# activate pygobject-2.14.1.lzm
# activate pygtk-2.12.1.lzm
Since I've started down the road to making the w3af module, lets go ahead and create a fakeroot and begin populating it for the final module build:
# mkdir fakeroot
# mkdir -p fakeroot/opt/owasp
# cp -a /opt/owasp/w3af fakeroot/opt/owasp/
# cp working/*.lzm ./
# lzm2dir pygobject-2.14.1.lzm fakeroot/
# lzm2dir pygtk-2.12.1.lzm fakeroot/
# find fakeroot | less
[check the progress of fakeroot]
# rm -rf fakeroot/var/
# rm -rf fakeroot/usr/share/gtk-doc/
# rm -rf fakeroot/usr/lib/pygtk/2.0/demos/
[remove some fluff]
OK. Those are the easy dependencies. Lets work on some that are a bit more involved - pysqlite from source. That said, lets first record what's already installed
# find / > /root/working/pre-w3af
# cd working
# wget http://oss.itsystementwicklung.de/download/pysqlite/2.5/2.5.0/pysqlite-2.5.0a.tar.gz
--2008-10-20 23:43:44-- http://oss.itsystementwicklung.de/download/pysqlite/2.5/2.5.0/pysqlite-2.5.0a.tar.gz
Resolving oss.itsystementwicklung.de... 88.198.116.118
[snip]
# tar -xzvf pysqlite-2.5.0a.tar.gz
# cd pysqlite-2.5.0
# python setup.py build
[bunch of happy messages]
# python setup.py install
running install
running build
[snip]
# python setup.py install
running install
running build
[snip]
# python
Python 2.5.2 (r252:60911, Mar 1 2008, 13:52:45)
[GCC 4.2.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pysqlite2 import test
>>> test.test()
...............................................................................
...............................................................................
----------------------------------------------------------------------
Ran 198 tests in 0.397s
OK
In theory, that's the last one I'll have to download as the rest come with the source. Lets do the installs that came with the source then figure out the diff from the find command we did above:
# cd /opt/owasp/w3af/extlib/fpconst-0.7.2/
# python setup.py install
running install
running build
[snip]
Writing /usr/lib/python2.5/site-packages/fpconst-0.7.2-py2.5.egg-info
# cd ../SOAPpy/
# python setup.py install
running install
running build
[snip]
Writing /usr/lib/python2.5/site-packages/SOAPpy-0.12.0-py2.5.egg-info
# cd ../cluster/
# python setup.py install
running install
running build
[snip]
Writing /usr/lib/python2.5/site-packages/cluster-1.1.1b3-py2.5.egg-info
# cd ../pyPdf/
# python setup.py install
running install
running build
[snip]
Writing /usr/lib/python2.5/site-packages/pyPdf-1.9-py2.5.egg-info
# cd ../pygoogle/
# python setup.py install
running install
running build
[snip]
creating /usr/lib/python2.5/site-packages/pygoogle.pth
# cd ../pywordnet/
# python setup.py install
running install
running build
[snip]
Writing /usr/lib/python2.5/site-packages/pywordnet-2.0.1-py2.5.egg-info
Beyond the great help of my anonymous benefactor, there are a few more Python modules which are missing from the Live CD 2008:
# cd /opt/owasp/w3af
# ./w3af_console
You have to install pyOpenSSL library. On Debian based distributions: apt-get install python-pyopenssl
# cd /root/working
# wget http://internap.dl.sourceforge.net/sourceforge/pyopenssl/pyOpenSSL-0.8.tar.gz
--2008-10-20 23:23:05-- http://internap.dl.sourceforge.net/sourceforge/pyopenssl/pyOpenSSL-0.8.tar.gz
Resolving internap.dl.sourceforge.net... 74.201.0.131
[snip]
# tar -xzvf pyOpenSSL-0.8.tar.gz
# cd pyOpenSSL-0.8
# python setup.py build
running build
running build_py
[snip]
# python setup.py install
running install
running build
[snip]
Writing /usr/lib/python2.5/site-packages/pyOpenSSL-0.8-py2.5.egg-info
# cd /opt/owasp/w3af/
# ./w3af_console
You won't be able to use the web20Spider without zc.testbrowser.real library installed. Exception: No module named
testbrowser.src.zc.testbrowser.real
global name 'Browser' is not defined. You can get MozRepl at http://hyperstruct.net/projects/mozlab .
Error while reading plugin options.
There was an error while importing plugins.output.gtkOutput: "No module named _sqlite3".
OK. This is the same crap I was getting on my first attempt. Once again pulling a line from my anonymous benefactor's script:
"wget -N -q http://nukeit.org/pub/20080730193723/_sqlite3.so && cp _sqlite3.so /usr/lib/python2.5/sqlite3/"
# find / -type f -name "*_sqlite3.so*"
[returns nothing]
I'm going to pull the same file from my 'real' Linux box (Ubuntu 8.04 - Hardy):
$ find / -type f -name "*_sqlite3.so*"
/usr/lib/python2.5/lib-dynload/_sqlite3.so
[scp this over to Live CD or use the method of your choice]
Now back on the running Live CD:
# _sqlite3.so /usr/lib/python2.5/sqlite3/ # cd /opt/owasp/w3af/ # ./w3af_console You won't be able to use the web20Spider without zc.testbrowser.real library installed. Exception: No module named testbrowser.src.zc.testbrowser.real global name 'Browser' is not defined. You can get MozRepl at http://hyperstruct.net/projects/mozlab . Error while reading plugin options. There was an error while importing plugins.output.gtkOutput: "/usr/lib/python2.5/sqlite3/_sqlite3.so: wrong ELF class: ELFCLASS64".
{Your choice of expletive here}! That's what I get for having a fancy 64 bit computer. OK. I'm going to grab one from a 32 bit Ubuntu 8.04 box I've got handy. I won't bother recording these steps. If I was thinking, I'd have never got that message. Back on the running Live CD:
# cp -i _sqlite3.so /usr/lib/python2.5/sqlite3/
cp: overwrite `/usr/lib/python2.5/sqlite3/_sqlite3.so'? y
# cd /opt/owasp/w3af/
# ./w3af_console
[snip]
Error while reading plugin options.
There was an error while importing plugins.output.gtkOutput: "/usr/lib/python2.5/sqlite3/_sqlite3.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8".
OK. Grasp at straws time. Lets look at what my install of pysqlite installed:
# ls /usr/lib/python2.5/site-packages/pysqlite2/
__init__.py __init__.pyc _sqlite.so* dbapi2.py dbapi2.pyc dump.py dump.pyc test/
[ I wonder if that _sqlite.so will work... ]
# cp -i /usr/lib/python2.5/site-packages/pysqlite2/_sqlite.so /usr/lib/python2.5/sqlite3/_sqlite3.so
cp: overwrite `/usr/lib/python2.5/sqlite3/_sqlite3.so'? y
# ./w3af_console
[snip]
Error while reading plugin options.
There was an error while importing plugins.output.gtkOutput: "dynamic module does not define init function (init_sqlite3)".
Well. This is going to kill me cause I just hate not figuring out why something didn't work but the late hour and the proxmity to the OWASP EU Summit 2008 makes me take a bit of a short cut. Again, props to my anonymous benefactor.
# cd /root/
# mv _sqlite3.so bad_sqlite3.so
# wget http://nukeit.org/pub/20080730193723/_sqlite3.so
--00:32:57-- http://nukeit.org/pub/20080730193723/_sqlite3.so
[snip]
# file _sqlite3.so
_sqlite3.so: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), not stripped
# strings _sqlite3.so | less
# strings _sqlite3.so > downloaded
# strings bad_sqlite3.so > ubuntu
# diff downloaded ubuntu | less
[ OK. My paranoia is satisfied there's nothing wonky with that file ]
# cp -i _sqlite3.so /usr/lib/python2.5/sqlite3/
cp: overwrite `/usr/lib/python2.5/sqlite3/_sqlite3.so'? y
# cd /opt/owasp/w3af/
# ./w3af_console
You won't be able to use the web20Spider without zc.testbrowser.real library installed. Exception: No module named testbrowser.src.zc.testbrowser.real
global name 'Browser' is not defined. You can get MozRepl at http://hyperstruct.net/projects/mozlab .
w3af>>> version
w3af - Web Application Attack and Audit Framework
Version: beta7
Revision: 1903
Author: Andres Riancho and the w3af team.
w3af>>> exit
w3af>>>
GPL v2 inside.
root@slax:/opt/owasp/w3af#
Excuse me for a bit while I do my happy dance...
OK. I've regained my composure. Now to try the GUI:
# ./w3af_gui You have to install pyparsing library. On Debian based distributions: apt-get install python-pyparsing
This appears to be a bug. It is also insisting on the pydot Python module which seems to be a bug as well. I've sent an email to Andres and a patch. I'm patching my source assuming it is correct. Moving on, I found out I needed Graphviz for the GUI. I handled that by Making the Graphviz module. Next up is pycairo which is apparently needed for pygtk. Time to add that to the mix:
$ wget http://slackware.mirrors.tds.net/pub/slackware/slackware-12.1/slackware/l/pycairo-1.4.12-i486-1.tgz
$ wget http://slackware.mirrors.tds.net/pub/slackware/slackware-12.1/slackware/l/pycairo-1.4.12-i486-1.tgz.asc
$ gpg --verify pycairo-1.4.12-i486-1.tgz.asc pycairo-1.4.12-i486-1.tgz
gpg: Signature made Tue 08 Jan 2008 10:53:37 PM CST using DSA key ID 40102233
gpg: Good signature from "Slackware Linux Project <security@slackware.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: EC56 49DA 401E 22AB FA67 36EF 6A44 63C0 4010 2233
[move the tarball over to the Live CD]
# cd working/
# tgz2lzm pycairo-1.4.12-i486-1.tgz pycairo-1.4.12.lzm
# cp pycairo-1.4.12.lzm ../
# cd ../
# lzm2dir pycairo-1.4.12.lzm fakeroot/
# activate pycairo-1.4.12.lzm
# cd /opt/owasp/w3af
# ./w3af_gui
Starting w3af, running on:
Python version:
2.5.2 (r252:60911, Mar 1 2008, 13:52:45)
[GCC 4.2.3]
GTK version: 2.12.9
PyGTK version: 2.12.1
w3af - Web Application Attack and Audit Framework
Version: beta7
Revision: 1903
Author: Andres Riancho and the w3af team.
WHOOT! You can't see it from the above, but the GUI launched! Happy Dance!
Now that that's over, lets do a find, diff and create a .lzm from fakeroot:
# find / > working/post-w3af
# cd working
# diff pre-w3af post-w3af > w3af-install
# vi w3af-install
[remove any cruft]
Use this file to determine what needs to be moved over to /root/fakeroot/ such as /opt/owasp/w3af and any python modules created above.
Create the needed directories in fakeroot
Eventually, we'll use dir2lzm to make the module, so lets get the directories created for menu items and such that we'll need later:
$ mkdir -p ./fakeroot/usr/bin $ mkdir -p ./fakeroot/usr/share/aplications $ mkdir -p ./fakeroot/usr/share/pixmaps
Create and put the module's files into the fakeroot directory
We'll need some scripts to start the various bits of w3af in fakeroot/usr/bin. The first two are pretty easy:
# vi fakeroot/usr/bin/w3af-cli [create script] # cat fakeroot/usr/bin/w3af-cli #!/bin/sh /opt/owasp/w3af/w3af-console "$@" # chmod 775 fakeroot/usr/bin/w3af_console # vi fakeroot/usr/bin/w3af_gui [create script] # cat fakeroot/usr/bin/w3af_gui #!/bin/sh /opt/owasp/w3af/w3af_gui "$@" # chmod 775 fakeroot/usr/bin/w3af_console
Because parts of w3af are command line tools, we're going to create two startup scripts to be used by the menu items for w3af_console
$ vi fakeroot/usr/bin/startup-w3af_console [create script] $ cat fakeroot/usr/bin/startup-w3af_console #/bin/sh echo " " echo " " echo " w3af -Web Application Attack and Audit Framework" echo " w3af_console - interactive w3af console" echo " (part of the OWASP Live CD 2008)" echo " " echo " _____ __ " echo " __ _|___ / __ _ / _|" echo ' \ \ /\ / / |_ \ / _` | |_ ' echo " \ V V / ___) | (_| | _|" echo ' \_/\_/ |____/ \__,_|_| ' echo " " echo " Type 'w3af_console --help' to get help " echo " or get started with 'w3af_console" echo " "
Creating w3af's two menu item is a bit more complex since one of them opens in a terminal. Use a text editor and create the files:
w3af_console.desktop
$ vi fakeroot/usr/share/applications/w3af_console.desktop [create the file] $ cat fakeroot/usr/share/applications/w3af_console.desktop [Desktop Entry] Categories=Application;Network; Comment= Encoding=UTF-8 Exec=startup-w3af_console; bash GenericName=w3af_console Icon=/usr/share/pixmaps/w3af-icon.png MimeType=text/html Name=w3af Interactive Console Path= StartupNotify=false Terminal=1 TerminalOptions=-T "w3af - Interactive Console" Type=Application X-KDE-StartupNotify=true X-KDE-SubstituteUID=false X-KDE-Username=
and w3af_gui-start.desktop
$ vi fakeroot/usr/share/applications/w3af_gui.desktop [create the file] $ cat fakeroot/usr/share/applications/w3af_gui.desktop [Desktop Entry] Encoding=UTF-8 Exec=w3af_gui Icon=/usr/share/pixmaps/w3af-icon.png Type=Application Categories=Application;Network; Name=w3af_gui Name[cs]=GUI for w3af GenericName=Start w3af_gui GenericName[cs]=Start w3af_gui MimeType=text/html X-KDE-StartupNotify=true
For the icon, I ripped off the owl from the project website. Did a little Gimp-foo and then moved that into fakeroot.
$ cp temp/w3af-icon.png fakeroot/usr/share/pixmaps/w3af-icon.png
w3af is rapidly developing as we speak (or you read this, I guess). I want you to be able to update to the latest svn code. Luckily, I've created a subversion client module. You can read about Making the Subversion client module if your curious.
As an added bonus, lets create an easy way to update your w3af install. First a script to actually do that:
$ vi fakeroot/usr/bin/update-w3af [create script] $ cat fakeroot/usr/bin/update-w3af #!/bin/sh cd /opt/owasp/w3af/ svn update
and a script to all from the menu:
vi fakeroot/usr/bin/startup-update-w3af [create script] $ cat fakeroot/usr/bin/startup-update-w3af #/bin/sh echo " " echo " " echo " w3af -Web Application Attack and Audit Framework" echo " w3af_console - interactive w3af console" echo " (part of the OWASP Live CD 2008)" echo " " echo " _____ __ " echo " __ _|___ / __ _ / _|" echo ' \ \ /\ / / |_ \ / _` | |_ ' echo " \ V V / ___) | (_| | _|" echo ' \_/\_/ |____/ \__,_|_| ' echo " " echo " " echo " Subversion + w3af = Always hot web p0wnage. Always " echo " Get started with 'update-w3af" echo " " echo " A message like 'At revision XXXX' means you are up to date" echo " "
and a menu item too:
$ vi fakeroot/usr/share/applications/update-w3af.desktop [create the file] $ cat fakeroot/usr/share/applications/update-w3af.desktop [Desktop Entry] Categories=Application;Network; Comment= Encoding=UTF-8 Exec=startup-update-w3af; bash GenericName=update-w3af Icon=/usr/share/pixmaps/w3af-icon.png MimeType=text/html Name=Update w3af Path= StartupNotify=false Terminal=1 TerminalOptions=-T "w3af - Updating the goodness" Type=Application X-KDE-StartupNotify=true X-KDE-SubstituteUID=false X-KDE-Username=
Everything is in place to create the module, a quick final check:
find fakeroot fakeroot/ fakeroot/opt fakeroot/opt/owasp fakeroot/opt/owasp/w3af fakeroot/opt/owasp/w3af/webroot fakeroot/opt/owasp/w3af/webroot/rfip.txt ...
Generate the .lzm module
This is the easy part.
$ ./dir2lzm ./fakeroot w3af-svn-r2161.lzm
Add the modules to the ISO build directory
Also cake
$ cp -i w3af-svn-r2161.lzm ../contents/slax/base/ $ chmod 775 ../contents/slax/base/w3af-svn-r2161.lzm
Test the new module
I like scp'ing the new modules into a running Live CD and using the Module Manager to Add modules to a running system. I typically have the Live CD running in a VM while I create modules so its already up and ready. Don't forget about dependencies on other modules (python-2.5.2-i486-1.lzm, sqlite-3.6.4.lzm, graphviz-2.30.3.lzm, and optionally subversion-client-1.5.2.lzm for updating w3af) Get those here if you need them.
You can also gen a new ISO and run it in a VM of your choice.
$ cd ../contents/slax/ $ ./make_iso.sh /home/mtesauro/isos/new-owasp.iso
If anything doesn't work as expected, make the changes needed to ./fakeroot and try again.
SPECIAL BONUS Here's a script to add w3af to a running OWASP Live CD. Its been tested successfully on the SoC release (the most current). If you want the menu items to show up, you'll need to pay attention to the final output of the script. Enjoy!
#!/bin/sh # Get and install the dependencies first ## sqlite wget http://owasp-livecd-2008.googlecode.com/files/sqlite-3.6.4.lzm activate sqlite-3.6.4.lzm ## graphviz wget http://owasp-livecd-2008.googlecode.com/files/graphviz-2.30.3.lzm activate graphviz-2.30.3.lzm # Gratuitous ldconfig - cause it doesn't work without it (grr) ldconfig # Get and install w3af wget http://owasp-livecd-2008.googlecode.com/files/w3af-svn-r2161.lzm activate w3af-svn-r2161.lzm # Make the menu items show up - should work but doesn't (grr) /usr/bin/update-desktop-database -q echo "" echo "" echo "You should be good to go. Enjoy" echo "You'll have to run 'kmenuedit' then click the save button for the icons to appear" echo "Haven't quite figured that bit out yet" echo "Look in the Internet section for icons until I update the owasp-branding module" echo "Also, the commands 'w3af_console' and 'w3af_gui' should be in your path" echo ""
Just create this in the text editor (or copy and paste it from this site). If you're stumped for an editor, try kwrite. Don't forget to make it executable. Example below:
# kwrite
[copy and paste the code above from this site - save it as "install-w3af"]
# chmod u+x install-w3af
# ./install-w3af
[bunch of output]
Clean up and archive
Once you've got a working module, lets clean up a bit.
$ $ mkdir ./completed_modules/w3af $ mv w3af-svn-r2161.lzm ./completed_modules/w3af/ $ rm -rf ./temp/*
I usually delete anything under ./fakeroot also.
$ rm -rf ./fakeroot/*
