Manually¶
Install AndroLyze¶
# clone recursive to fetch submodules
git clone --recursive https://github.com/nachtmaar/androlyze.git
cd androlyze
# copy the sample config file
cp androlyze/settings/defaults/config.conf conf/
cp androlyze/settings/defaults/script_settings.json conf/
# copy config for google-play-crawler (needed for playstore.py)
cp google-play-crawler/googleplay/crawler.conf conf/
# revoke not needed permissions for config files (contains credentials)
chmod 600 -R conf/
Install other requirements¶
# install pip
sudo apt-get install python-pip
# install python requirements
pip install --user -r docker/worker/requirements.txt
Distributed Environment¶
# only for task server
sudo apt-get install rabbitmq-server
# only for result database
sudo apt-get install mongodb
MongoDB¶
Also see MongoDB installation
Authentication and SSL¶
Note
The user creation shown below is for MongoDB version 2.4, 2.2 is a different API ! For version 3 one has to use db.createUser instead of db.addUser but with the same arguments!
First we need to add an account. One for user administration, second for database r/w access.
# start the mongo shell
mongo <host>
use admin
# add useradmin
db.addUser( { user: "useradmin",
pwd: "pwd",
roles: [ "userAdminAnyDatabase"] } )
# add androlyze user (needs full access to create user supplied databases via config file)
db.addUser( { user: "androlyze",
pwd: "pwd",
roles: [ "readWriteAnyDatabase"] } )
Enable authentication in config file (/etc/mongodb.conf)
# mongodb.conf
# Where to store the data.
dbpath=/var/lib/mongodb
#where to log
logpath=/var/log/mongodb/mongodb.log
logappend=true
bind_ip = 127.0.0.1
port = 27017
# Enable journaling, http://www.mongodb.org/display/DOCS/Journaling
journal=true
auth = true
For X509 certificate creation have a look at RabbitMQ website. For SSL add these values too:
# SSL options
sslMode = requireSSL
# SSL Key file and certificate
sslPEMKeyFile = /etc/ssl/private/mongodb/mongodb.pem
sslOnNormalPorts = true
# ca certificate
sslCAFile = /etc/ssl/certs/androlyze_ca.pem
# client don't need a certificate
sslWeakCertificateValidation = true
Restart server
sudo /etc/init.d/mongodb restart
Warning
If you are encountering any troubles with starting the mongodb server, try to start it manually with mongod –config /etc/mongodb.conf and have a look at the log file.
or test with sudo -u mongodb mongod –config /etc/mongodb.conf
RabbitMQ¶
RabbitMQ configuration (see this for more details):
sudo rabbitmqctl add_user androlyze <pw>
sudo rabbitmqctl add_vhost androlyze_vhost
sudo rabbitmqctl set_permissions -p androlyze_vhost androlyze ".*" ".*" ".*"
Extra space¶
We need a big task storage, so if your root disk is not big enough use some different path. Default is /var/lib/rabbitmq.
Set in file “/etc/rabbitmq/rabbitmq-env.conf”
RABBITMQ_MNESIA_BASE=/custompath/lib/rabbitmq/
SSL¶
For X509 certificate creation have a look at RabbitMQ website.
File: /etc/rabbitmq/rabbitmq.config
[
{rabbit, [
{tcp_listeners, []},
{ssl_listeners, [5671]},
{ssl_options, [{cacertfile,"/etc/ssl/certs/androlyze_ca.pem"},
{certfile,"/etc/ssl/certs/androlyze_server.pem"},
{keyfile,"/etc/ssl/private/rabbitmq/androlyze_server.key"},
{verify,verify_peer},
{fail_if_no_peer_cert,true}]}
]}
].
Warning
For errors have a look at the log file:
tail -n 50 /var/log/rabbitmq/rabbit....log
Deployment and Management¶
AndroLyze follows a classical cluster design. Therefore our deployment and manegement is automized with the help of Fabric which uses ssh to connect to the workers and executes the tasks locally on them.
Therefore you need to do set a few values in the Deployment section of the distributed config file.
Available fabric commands:
cnt_processes Cnt "celery worker" processes
deploy_project If `user` and/or `passwd` given, use them to authenticate when cloning via http.
deploy_scripts Deploy the scripts on the workers. `scripts_src` is the script folder what shall be synced
deploy_testing Deploy the testing code on the workers. Intended for usage where changes have not been comitted (or not to master)
initial_worker_setup Initial worker setup. Needs root access.
install_dependencies Set up the workers
kill_processes Kill the workers by sending the kill signal.
list_processes List "celery worker" processes
restart_workers Restart workers on registered hosts with specified concurrency.
start_workers Start workers on registered hosts with specified concurrency.
stop_workers Stop all workers.
# uses the username specified in config file or your username
fab start_workers
# don't try public-key auth
fab start_workers -k
# use worker as username
# otherwise
fab start_workers -u worker
# you can also use hosts other than in the config file
fab start_workers -H user@worker
# restart workers with default concurrency=#number of cores
fab restart_workers
# custom conurrency
fab restart_workers:concurrency=16
# use autoscaling, min: #number of cores, max: min * multiplicator
fab restart_workers:autoscale=True,autoscale_mult=2
# kill workers
fab kill_processes
# deploy scripts on workers
fab deploy_scripts:scripts_user/nils/
Warning
# don't use! it will wipe the scripts from other users!
fab deploy_scripts:scripts_user