Cronus Table of Contents¶

Quick Setup¶

Steps to install Cronus agent and Cronus master on a machine.

Advance Setup¶

# cronusmaster public and private IP, needed for smart parameters in command
agentmaster.externalIp=cronusmaster_external_ip
agentmaster.internalIp=cronusmaster_internal_ip

# user data location (applicable only if use local FS for user data)
agentmaster.userDataDao.file.dir=.appdata

# agent basic authentication, need this if remote agent auth is enabled
agentmaster.cronusagent.password=username:password

# provide path_to_ssl_cert for custom ssl cert
# set user:password for basic authentication
wget -qO- 'http://cronuspaas.github.io/downloads/install_agent' \
| sudo server_pem=path_to_ssl_cert agent_pwd=user:password bash

Deploy Application Workflow¶

This document describe steps to deploy an application to cloud by using Cronus framework.

Introduction¶

Cronusmaster can execute any http(s) request against one or many target nodes. Think of it as a RESTful client that can run with predefined template on target node grammatically at scale. Cronusmaster is used to execute different tasks on cronus agent through its RESTful interface.

• Command: Defines a HTTP(S) request template
• Nodegroup: Defines a list of nodes that will be targets of a command
• Job: A command that runs on a nodegroup
• Scheduled Job: A command run on a nodegroup with a recurring schedule
• Log: Execution result and log of a job

To put cronusmaster to work

1. Define HTTP(S) command template
2. Define nodegroup with target node IPs or hostnames
3. Execute materialized commmand on a nodegroup as a job
4. Check result of the job in the log
5. Optinoally create an oneclick launch from an already executed job
6. Optionally create a recurring job from an already executed job or oneclick launch

cronusmaster main page

Command Template¶

Command template is a complete HTTP(S) request in json format with replaceable variables

Sample synchronous HTTP request

{
 "name" : "SAMPLE_SYNC_COMMAND",                              // unique name of the command
 "category" : "sample",                                       // optional category name for grouping
 "httpTaskRequest" : {
   "taskType" : "async",                                      // this is a http request with response
   "httpClientType" : "httpClient",                           // type of http client to use
   "executionOption" : {                                      // below are options for sending request
     "timeOutSec" : 0,                                        // time out in second
     "initDelaySec" : 0,                                      // initial delay in second
     "maxRetry" : 3,                                          // total # of retry on failure
     "retryDelaySec" : 0                                      // delay before retry
   },
   "urlTemplate" : {                                          // http request template
     "url" : "https://<host>:19000/services/<serviceName>",   // url
     "method" : "POST",                                       // http method
     "body" : {},                                             // http body
     "headers" : {
       "content-type" : "application/json"                    // http headers
     },
     "parameters" : {}                                        // query string parameters
   },
   "resProcessorName" : "agentProcessor"                      // bean for processing http response
 },
 "userData" : {                                               // user provided data to launch command Job
  "serviceName" : "myservice"
 }
}

Sample asynchronous HTTP request with polling for result

{
 "name" : "SAMPLE_ASYNCPOLL_COMMAND",
 "category" : "sample",
 "httpTaskRequest" : {
   "taskType" : "asyncpoll",                              // this is a request followed by polling
   "httpClientType" : "httpClient",
   ...                                                    // same url template as above for request
   "monitorOption" : {                                    // options for polling
     "timeOutSec" : 0,
     "initDelaySec" : 0,
     "maxRetry" : 3,
     "retryDelaySec" : 0,
     "intervalSec" : 10                                   // polling interval in second
   },
   "pollTemplate" : {                                     // http polling template
     "url" : "https://<host>:19000/status/<uuid>",
     "method" : "GET",
     "body" : {},
     "headers" : {
       "content-type" : "application/json"
     },
     "parameters" : {}
   },
   "resProcessorName" : "agentPollProcessor",             // bean for processing http response
 },
 "userData" : {
   "serviceName" : "myservice"
 }
}

Sample agent request

{
  "name" : "Agent_Clean_Service",
  "category" : "agent",              // agent command with many fields predefined and can be skipped
  "httpTaskRequest" : {
    "taskType" : "asyncpoll",
    "httpClientType" : "httpClient",
    "urlTemplate" : {
      "url" : "https://<host>:19000/services/<serviceName>/action/cleanup",
      "method" : "POST",
      "body" : { },
      "headers" : { },
      "parameters" : { }
    }
  },
  "userData" : {
    "serviceName" : "myservice"
  }
}

Command¶

Create command

1. Command can be created in two ways
   • Create from scratch
   • Clone from an existing command
2. Define command type
   • async: request with reponse as the result
   • asyncpoll: request followed by polling to get result
3. Fill in the command template with http request values, similar to defining http request in RESTful client like POSTMAN
4. Define template parameter; template parameter can be used in all http reuqest values, template parameter is identified by “<>”, minimally if the command can be executed against different hosts, “<host>” needs to be part of the url value. Template parameters will be replace with real values at execution time, or if values are not provided, the parameters will be skipped.
5. Define user data, it will be used to drive command launch wizard

edit command

Modify command

Except the name, which is immutable, data of the command can be modified

Delete command

Delete command from persistent store

command summary page

Run command

Run command will launch command wizard

• wizard screen 1: customize command with execution options, user data
• wizard screen 2: select nodegroup against which command will be run
• wizard screen 3: confirm and launch command job, job progress and result can be found in command job log.

wizard screen 1

wizard screen 2

wizard screen 3

OneClick Launch¶

Oneclick launch captures all the input necessary to launch a command job, this allows a job to be launched with “one click” (or one API call).

Why it is useful

• Deploy or rollback applications with one click or one API call
• Save a job for repeated execution later
• Enhance automation with adaptive smart template parameters, for example can deploy latest application package to target nodes as the application package gets uploaded

Sample oneclick launch

{
 "name": "deploy_pyserver_local",                      // unique name of the oneclick command
 "userData": {                                         // custom user data
   "exe_retry": "3",                                   // execution options
   "mon_retry": "3",
   "thrStrategy": "UNLIMITED",
   "mon_int": "10"
   // user data
   "var_values": "{\"serviceName\":\"pyserver\",\"package\":[\"http://<cmInternalIp>:9000/agent/downloadPkg/<pyserver-.*.all.cronus.latest>\"]}",
 },
 "commandKey": "Agent_Deploy_Service",                 // command name
 "nodeGroupKey": "LOCALHOST"                           // nodegroup name
}

Create oneclick launch

Oneclick launch can be created in two ways

1. From command log tab, save an already executed command job as oneclick launch
2. From oneclick launch tab, create by cloning an existing oneclick launch

Smart template parameters

The following smart template parameters can be used in oneclick launch to allow greater degree of reusability

• <cmExternalIp>: reference public IP of the cronus master, value defined in cronusmaster config file
• <cmInternalIp>: reference private IP of the cronus master, value defined in cronusmaster config file
• <service-.*.platform.cronus.latest>: reference latest uploaded package for a service, for example <pyserver-.*.all.cronus.latest> is the latest version of the pyserver package, <pyserver-1.*.all.cronus.latest> is the latest version of 1.x.y of the pyserver package

With smart template parameters, one can create an oneclick launch to deploy latest application package to a group of nodes, without having to change anything in the oneclick launch.

Run oneclick

Run from “oneclick launch” tab with a single click, or one API call, redirect to command job log page upon successful launch.

oneclick launch summary

Nodegroup¶

Nodegroup defines a group of nodes with similar characteristics, and similar purpose. For example, a group of nodes that install the same operating system and cronus agent, in the same network, and are running the same application.

Create nodegroup

Nodegroup is uniquely identified by its name, values are line separated hostnames or IPs.

Use nodegroup

• Can be referenced in command wizard when run command job
• Can be referenced in oneclick launch
• Can be referenced in recurring job

nodegroup summary

edit nodegroup

Recurring Job¶

An existing one time command job or oneclick launch can be made to run repeatly as a recurring job.

recurring jobs

Create Recurring Job

1. Recurring job can be created in two ways
   • Create from an existing command job from command job log
   • Create from an existing oneclick launch
2. Go through the recurring job wizard to define a name uniquely identified the job, and recurring frequency in the interval of 5 minutes
3. Once saved, job running state can be toggled on and off from the Job page
4. Job log of previously launched job can be found in job log

job wizard screen 1

job wizard screen 2

Command Log¶

Log provide important information about result of command job, and provide chance to create new oneclick launch and recurring job.

Information available from the log

• status: status of the job (running|success|failure)
• S:F:Other: number of nodes for each status value
• input: customize user data
• result: summary of the result in json format, the raw response form the target node are truncated to 200 characters
• logs: full text search of the job result, as well as raw logs from the target node
• rerun: run the same command job again
• oneclick: create a new oneclick launch from the command job
• schedule: create a recurring job from the command job
• remove: delete the particular log, as well as related full text search entry

command logs

full text search

Configurations¶

This section explains cronus master configurations

Basic configurations

Cronus master configuration section in a sample play application.conf

# use local file | aws_s3 | openstack_swift to store user data
agentmaster.userDataDao=file
agentmaster.logProgressEnabled=true

# cronusmaster public and private IP if any
agentmaster.externalIp=some_external_ip
agentmaster.internalIp=some_internal_ip

# user data location (applicable only if use local fs for user data)
#agentmaster.userDataDao.file.dir=.appdata

# s3 related configs (applicable only if use aws_s3 for user data)
agentmaster.userDataDao.s3.myAccessKeyID=your_access_key_id
agentmaster.userDataDao.s3.mySecretKey=your_secret_key

# swift related configs (applicable only if use swift for user data)
agentmaster.userDataDao.swift.tenantId=your_tenant_id
agentmaster.userDataDao.swift.tenantName=your_tenant_name
agentmaster.userDataDao.swift.username=your_username
agentmaster.userDataDao.swift.password=your_password
agentmaster.userDataDao.swift.authenticationUrl=authn_url

# elastic search
# use embedded or remote elastic search backend
agentmaster.localEsEnabled=true
# location of search index (applicable only if use embedded elastic search)
agentmaster.esData=user_data/elasticsearch_data/data
# search end point (applicable only if use remote elastic search)
agentmaster.esEp=some_search_host

# agent basic authentication
agentmaster.cronusagent.password=username:password
# agent PKI authentication
agentmaster.cronusagent.pkicert=path_to_cert

# ... rest of the play configuration in application.conf

Configuration override for different environment

One can build separate application.conf.{environment} in the play conf/ directory to be used for deploying cronus master in different environment. Upon deployment time

1. Pass the environment value through agent deploy API call, e.g. curl -X POST '{..., "env": "prod"}' https://host:19000/services/cronusmaster/action/deploy
2. Agent will try to find the configuration file with .prod postfix and use it (by renaming it to application.conf)
3. Or if the environment specific configuration file is not found, will use the default base configuration application.conf

Same schemes work for logging configurations (log4j.properties) as well.

CronusMaster APIs¶

A few CronusMaster APIs can be called directly with json response.

• GET /commands/oneclickJson : list all oneclick launch
• GET /commands/oneclickRunJson?dataId={oneclick_launch_name} : execute oneclick launch

Additional APIs that drives CronusMaster UI can be found in play “routes” file.

Install in IaaS Cloud¶

Follow the steps to install agent in AWS EC2, or Goggle GCE Cloud

Prerequisites

• linux (Ubuntu, Cent, Redhat, Fedora etc.)
• sudo permission
• python > 2.6.7 && < 3
• openssl > 0.9.8
• wget
• system management daemon systemd or upstart

Install in dev (default ssl cert, no auth)

# install agent
wget -qO- 'http://cronuspaas.github.io/downloads/install_agent' | sudo dev=true bash
# check agent availability
curl -k https://localhost:19000/agent

Install in prod

# install agent
wget -qO- 'http://cronuspaas.github.io/downloads/install_agent' | \
sudo server_pem=path_to_ssl_cert agent_pwd=user:password bash
# check agent availability
curl -k https://localhost:19000/agent

Application Lifecycle¶

This section describes application lifecycle defined by Cronus framework.

Application Lifecycle Scripts¶

This guide describe how to create application lifecycle scripts.

1. Bootstrap your application with cronus package structure and lifecycle script skeleton

   cd application_root
   wget -qO- 'http://cronuspaas.github.io/downloads/bootstrap_cronus' | DIR=. bash
   

2. Modify scripts to your need

   • Startup script have nodaemon mode, distinguished by cmd line parameter $1=”nodaemon”

     • in nodaemon mode, application process is managed by external service manager (systemd or upstart), in nodaemon mode startup script needs to start the application and block until application exit.
     • in daemon mode, use exec or nohup to fork your application main process, also use setsid to detach your application process from parent agent process.

   • All parameters passed in from agent deploy API call are made available to lifecycle scripts as environment parameters except startup and shutdown scripts, reason being the two script can be executed multiple times out of context of a deployment.

   • Locally persisted application data that have lifecycle beyond the manifest can be accessed through .appdata symlink under the package root directory.

3. Test locally make sure scripts are working as expected

   • Optionally delete the scripts that are not needed

4. Package and upload, test deploy remotely in the cloud

Deploy Application¶

This guide describes how to deploy an application directly through agent REST API without cronusmaster

Requires:

• Cronus packaged application HTTP accessible by agent on target machine

How:

curl -k -X POST -d '{"package": ["{cronus_package_url}"], "manifest": "{manifest_name}", \
"env": "{env}", "daemon": "upstart|systemd"}' \
https://host:19000/services/{service_name}/action/deploy

# service_name: mandatory service name
# cronus_package_url: mandatory url location to your cronus package
# manifest_name: optional manifest name, default to use cronus package version
# env: optional env value, used to select environment specific application configurations
# daemon: optional daemon manager, used to launch applications managed by upstart or systemd

Expected Outcome:

• Service is created if not already exist
• Manifest is created if not already exist
• Any existing active manifest is shutdown, deactivate
• New manifest is install, activate, and startup

Application Data¶

This guide describes how to access locally persisted application data

Data local to specific manifest

Any data within the manifest package, or generated after activation in the manifest package is local to the specific manifest only. It can be deleted by agent after the manifest is deactivated and garbage collected

Data local to specific service

Agent creates .appdata under service root directory and creates symlink .appdata in each cronus package root during manifest activation. Data persisted in .appdata and its subdirectories is accessible across all manifests of the service. It can be deleted by agent when the service is deleted.

- service_root
    - .appdata
    - manifests
        - active
            - package1
                - .appdata (symlink to service_root/.appdata)

Environment variables

Any data passed in from deploy API will be available to the application life cycle script as environment variables

# deploy API call
    curl -k -X POST -d '{"package": ["http://$_cronus_package_url_"], "env": "production"' \
    https://host:19000/services/$_service_name_/action/deploy

# in activate script, this will print env=production
    echo "env=$env"

Basic Authentication¶

This guide describe

• How to install agent with SSL cert
• How to enable agent basic authentication

wget -qO- 'http://cronuspaas.github.io/downloads/install_agent' \
| sudo server_pem={path_to_pem_file} agent_pwd={user:password} bash

Expect outcome

• Agent up and running with custom SSL cert
• Agent CUD APIs requires http Basic authentication (HTTP header “Authorization: Basic {base64_encoded_user:password}”)

Agent Upgrade¶

This section describes how to upgrade an existing Agent

New agent package must exist in release folder in git http://github.com/cronuspaas/pkgs/{target_version}.unix.cronus

curl -k -H "content-type:application/json" -X POST -d '{"version": "version_eg_0.1.46"}' \
https://host:19000/agent/selfupdate

or

Update through cronusmaster UI

Expect Outcome:

Agent upgraded to new version, version number can be found in agent VI page https://host:19000/agent/ValidateInternals.html

Security Best Practice¶

Why:

• Agent has elevated access to resource required by its workload
• Agent is mission critical service, interference is not desirable
• Agents are deployed everywhere in target infrastructure

Existing Agent Security Mechanisms:

• All traffic through agent API on HTTPS
• Basic HTTP authentication or PKI based authentication
• Separation of agent user and app user, application managed by agent does not have same privilege as agent has
• Agent and app users are setup as system accounts with login disabled

Security Best Practice:

• Do not enable access to agent API from internet without any restriction. It’s best to only allow access from intranet via cronusmaster, or if that’s not feasible, only allow access from internet from trusted IPs
• Do NOT use default SSL certs when install agent, always provide your own certs
• Do NOT leave agent API open without authentication, at least set basic auth with your own user:password
• Do NOT keep your credentials in your source code, or encrypt it with your private key

Agent API List¶

• “GET /services/” : list all services
• “GET /services” : list all services
• “POST /services/{service}” : create service
• “DELETE /services/{service}” : delete service
• “GET /services/{service}” : list manifests of service
• “GET /services/{service}/action/log” : view service log
• “POST /services/{service}/action/updatemetadata” : set service metadata
• “GET /services/{service}/action/getmetadata” : get service metadata as json
• “GET /services/{service}/action/getmetadataprops” : get service metadata as property format
• “GET /services/{service}/action/listpackages” : list all packages of the service
• “POST /services/{service}/manifests/{manifest}” : create manifest
• “DELETE /services/{service}/manifests/{manifest}” : delete manifest
• “GET /services/{service}/manifests/{manifest}” : list all manifests
• “GET /services/{service}/manifests/{manifest}/action/log” : view manifest log
• “POST /services/{service}/manifests/{manifest}/action/activate” : activate manifest
• “POST /services/dist/startdownload” : download cronus package
• “POST /services/dist/validatepackage” : validate md5sum of cronus package against its prop file
• “GET /services/dist/listpackages” : list all packages
• “POST /services/dist/uploadpackage” : upload a cronus package
• “DELETE /services/dist/{package}” : delete a package
• “GET /status/{uuid}” : get status of a job (from job uuid)
• “DELETE /status/{uuid}” : cancel a job (from job uuid)
• “GET /status/uuidoutput/{uuid}” : get raw logs of a job (from job uuid)
• “GET /status/guidoutput/{guid}” : get raw logs of a job (from pass in guid)
• “GET /agent/logs” : view agent logs
• “GET /agent/logs/{fileName}” : view one agent log file
• “GET /agent” : view agent ValidateInternals page
• “GET /agent/ValidateInternals” : agent ValidateInternals in json format
• “GET /agent/ValidateInternals.html” : agent ValidateInternals html page
• “GET /agent/agenthealth” : agent health information in json format
• “GET /agent/monitors” : agent run monitoring value snapshot
• “GET /agent/threads” : agent thread information in json format
• “POST /services/{service}/action/restart” : restart service
• “POST /services/{service}/action/reset” : reset current active manifest (shutdown, deactivate, activate, startup)
• “POST /services/{service}/action/startup” : startup service
• “POST /services/{service}/action/shutdown” : shutdown service
• “POST /services/{service}/useraction/{useraction}” : run user provide script
• “POST /services/{service}/action/deactivatemanifest” : deactivate current active manifest
• “POST /services/{service}/action/deploy” : deploy a service, create service, manifest and activate
• “POST /services/{service}/action/rollback” : rollback a service to last activated version
• “DELETE /services/{service}/action/cleanup” : delete everything of a service
• “POST /agent/cleanup” : delete all services except agent from the host
• “POST /agent/shutdown” : shutdown agent
• “POST /agent/safeshutdown” : shutdown agent when agent is idle (no activity)
• “POST /agent/selfupdate” : upgrade agent
• “DELETE /agent/selfupdate” : cancel an agent upgrade
• “GET /agent/config” : get agent config
• “POST /agent/config” : override agent config
• “DELETE /agent/config” : delete an agent config override
• “POST /agent/cleanproc” : cleanup all cronusapp processes
• “GET /agent/api” : list this agent API
• “POST /agent/gc” : ondemand agent GC
• “GET /agent/servicesinfo” : list all services and everything of the services
• “GET /agent/modules” : get list of agent modules
• “GET /agent/modules/{module}” : get information of a agent module
• “POST /agent/modules/{module}” : create an agent module
• “DELETE /agent/modules/{module}” : delete an agent module
• “POST /admin/executeScript” : execute script
• “POST /admin/executeCmd” : execute a shell command
• “GET /security/listkeys” : list PKI keys
• “POST /security/key/{key}” : create PKI key
• “DELETE /security/key/{key}” : delete PKI key
• “POST security/validatetoken” : validate an existing token

Agent Configuration¶

Agent configurations and default values, configurations can be override through API call

Agent Configurations

agent_health_report_interval=120               # interval when agent health is checked
agent_thread_timeout=600                       # agent thread timeout
app_restart_init_delay=15                      # artificial delay before agent starts services after system reboot
download_skip_prop=true                        # not require cronus prop file
download_thread_attempt=1                      # download retry
download_thread_max_time=7200                  # download timeout ceiling
download_thread_min_progress_time=10           # download progress timeout
download_thread_min_time=60                    # download timeout floor
download_thread_rate_per_sec=102400            # download rate throttle
exec_thread_progress_timeout=60                # task execution thread progress timeout
exec_thread_sleep_time=1                       # interval when check task execution progress
exec_thread_timeout=600                        # task execution thread timeout
health_agent_file_descriptor_usage_threshold=500 # max number of file descriptor
health_agent_mem_usage_threshold=200000        # max memory
health_disk_usage_gc_buffer=20                 # agent disk percentage for gc
health_disk_usage_gc_threshold=70              # agent disk percentage threshold for gc
health_disk_usage_percent_threshold=90         # max disk usage percentage
keep_past_manifests=3                          # how many past manifest to keep
monitor_check_timeinterval=120                 # application monitoring interval
monitor_probation_enabled=true                 # probation for application monitoring script
monitor_publish_enabled=false                  # whether to publish application monitoring data
packageMgr_garbage_freq=1800                   # agent gc interval in seconds
packageMgr_install_package_age=43200           # agent package gc max age for installed package
packageMgr_install_package_min_age=7200        # agent package gc min age for installed package (shorter than this is not subject to gc)
packageMgr_package_age=86400                   # agent package gc max age for downloaded package
packageMgr_package_min_age=7200                # agent package gc min age for downloaded package (shorter than this is not subject to gc)
pkiauth_enabled=false                          # pki auth
safeshutdown_busy_check_timeout=86400          # timeout checking for safe shutdown
skip_cleanup_on_failure=false                  # skip cleanup when deploy application failure
system_restart_time_threshold=300              # system uptime check for system reboot
threadMgr_garbage_freq=60                      # agent thread gc interval
threadMgr_thread_age=3600                      # agent thread age for gc

Override Configuration

curl -k -X POST -d '{"configs": {"<configName>": "<configValue>"}}' https://host:19000/agent/config

Setup Cronus Agent¶

Setup Cronus Master¶

Change CronusMaster Configurations¶

How to change CronusMaster configurations and redeploy

Key take away

• Change configurations to enable variable used in one click command
• Change configurations to enable agent password based security
• Deploy CronusMaster with environment specific configuration file

Deploy App to Remote Compute¶

How to create a new nodegroup with a remote compute in CronusMaster, and deploy an app to it.

In the setup

• CronusMaster
• A remote compute with agent installed

Build and Package Cronus Package¶

How to build and package a cronus enabled node application and deploy through CronusMaster

• Checkout cronusstacks from git, http://github.com/cronuspaas
• Run download_container.sh to download node runtime (adjust the script for the different node runtime of your choice, and additional node module installation)
• Make sure env variable $cronusmaster=host is set
• Run “package.sh upload” to package and upload the package (package.sh alone will package cronus deployable package)
• Deploy in CronusMaster (look for uploaded package in Agent -> Package section), check app running at http://localhost:1337