EC2 Service Configuration 3.4

{{INLINETOC}} ====== Overview ====== The OpenNebula EC2 Query is a web service that enables you to launch and manage virtual machines in your OpenNebula installation through the [[http://docs.amazonwebservices.com/AWSEC2/2009-04-04/DeveloperGuide/index.html?using-query-api.html | Amazon EC2 Query Interface]]. In this way, you can use any EC2 Query tool or utility to access your Private Cloud. The EC2 Query web service is implemented upon the **OpenNebula Cloud API** (OCA) layer that exposes the full capabilities of an OpenNebula private cloud; and [[http://www.sinatrarb.com/ | Sinatra]], a widely used light web framework. {{ documentation:rel1.4:econe-arch_v2.png?400 |}} The current implementation includes the basic routines to use a Cloud, namely: image upload and registration, and the VM run, describe and terminate operations. The following sections explain you how to install and configure the EC2 Query web service on top of a running OpenNebula cloud.

:!: The OpenNebula EC2 Query service provides a Amazon EC2 Query API compatible interface to your cloud, that can be used alongside the native OpenNebula CLI or the libvirt interface.

:!: The OpenNebula distribution includes the tools needed to use the EC2 Query service.

====== Requirements & Installation ====== You must have an OpenNebula site properly configured and running , be sure to check the [[.#designing_and_installing_your_cloud_infrastructure|OpenNebula Installation and Configuration Guides]] to set up your private cloud first. This guide also assumes that you are familiar with the configuration and use of OpenNebula. The OpenNebula EC2 Query service was installed during the OpenNebula installation, and the dependencies of this service are installed when using the install_gems tool as explained in the [[.:ignc#ruby_libraries_requirements_front-end|installation guide]] If you installed OpenNebula from source you can install the EC2 Query dependencias as explained at the end of the [[.:compile#ruby_dependencies-end|Building from Source Code guide]] ====== Configuration ====== The service is configured through the ''/etc/one/econe.conf'' file, where you can set up the basic operational parameters for the EC2 Query web service. The following table summarizes the available options:

^ VARIABLE ^ VALUE ^ ^ Server configuration ^^ |:one_xmlrpc| oned xmlrpc service, http://localhost:2633/RPC2| |:server | Host where econe server will run | |:port | Port where econe server will run | ^ Log ^^ |:debug_level | Log debug level, 0 = ERROR, 1 = WARNING, 2 = INFO, 3 = DEBUG | ^ Auth ^^ |:auth | Authentication driver for incomming requests | |:core_auth | Authentication driver to communicate with OpenNebula core. Check [[.:cloud_auth | this guide]] for more information about the core_auth system | ^ Resources ^^ |:instance_types | The VM types for your cloud | |:datastore_id | DDatastore in which the Images uploaded through EC2 will be allocated, by default 1 | |:cluster_id | Cluster associated with the EC2 resources, by default no Cluster is defined | ^ Elastic IP ^^ |:elasticips_vnet_id | VirtualNetwork containing the elastic ips to be used with EC2. If no defined the Elastic IP functionality is disabled | |:associate_script | Script to associate a public IP with a private IP arguments: elastic_ip private_ip vnet_template(base64_encoded) | |:disassociate_script | Script to disassociate a public IP arguments: elastic_ip |

:!: The '':server'' **must** be a FQDN, do not use IP's here.

:!: Preserve YAML syntax in the ''econe.conf'' file.

Example:


#############################################################
# Server Configuration
#############################################################

# OpenNebula sever contact information
:one_xmlrpc: http://localhost:2633/RPC2

# Host and port where econe server will run
:server: localhost
:port: 4567

# SSL proxy that serves the API (set if is being used)
#:ssl_server: fqdm.of.the.server

#############################################################
# Auth
#############################################################

# Authentication driver for incomming requests
#   ec2, default Acess key and Secret key scheme
#   x509, for x509 certificates based authentication
:auth: ec2

# Authentication driver to communicate with OpenNebula core
#   cipher, for symmetric cipher encryption of tokens
#   x509, for x509 certificate encryption of tokens
:core_auth: cipher

#############################################################
# Log
#############################################################

# Log debug level
#   0 = ERROR, 1 = WARNING, 2 = INFO, 3 = DEBUG
:debug_level: 3

#############################################################
# Resources
#############################################################

# Cluster associated with the EC2 resources, by default no Cluster is defined
#:cluster_id:

# Datastore in which the Images uploaded through EC2 will be allocated, by default 1
#:datastore_id:

# VM types allowed and its template file (inside templates directory)
:instance_types:
  :m1.small:
    :template: m1.small.erb

#############################################################
# Elastic IP
#############################################################

# VirtualNetwork containing the elastic ips to be used with EC2. If no defined
#   the Elastic IP functionality is disabled
#:elasticips_vnet_id: 

# Script to associate a public IP with a private IP
#   arguments: elastic_ip private_ip vnet_template(base64_encoded)
:associate_script: /usr/bin/false

# Script to disassociate a public IP
#   arguments: elastic_ip
:disassociate_script: /usr/bin/false

===== Cloud Users ===== The cloud users have to be created in the OpenNebula system by ''oneadmin'' using the ''oneuser'' utility. Once a user is registered in the system, using the same procedure as to create private cloud users, they can start using the system. The users will authenticate using the [[http://docs.amazonwebservices.com/AWSEC2/latest/DeveloperGuide/index.html?using-query-api.html|Amazon EC2 procedure]] with ''AWSAccessKeyId'' their OpenNebula's username and ''AWSSecretAccessKey'' their OpenNebula's hashed password. The cloud administrator can limit the interfaces that these users can use to interact with OpenNebula by setting the driver "public" for them. Using that driver cloud users will not be able to interact with OpenNebula through Sunstone, CLI nor XML-RPC. $ oneuser chauth cloud_user public ===== Networking for the Cloud VMs ===== By default, the templates includes a ''NIC'' interface to be attached to a virtual network. You **have to create this network** using the ''onevnet utility'' with the IP's you want to lease to the VMs created with the EC2 Query service. $ onevnet create /tmp/templates/vnet ID: 4 Remember that you will have to add this VNet to the users group (ID:1) and make it public in order to get leases from it. $ onevnet chgrp 4 1 $ onevnet publish 4

:!: You will have to update the NIC template, inside the ''/etc/one/ec2query_templates'' directory, in order to use this VNet ID

===== Defining VM Types ===== You can define as many Virtual Machine types as you want, just: * Create a template for the new type and place it in ''/etc/one/ec2query_templates''. This template will be //completed// with the data for each cloud //run-instance// request, and then submitted to OpenNebula. You can start by modifying the ''m1.small.erb'' example, to adjust it to your cloud:


NAME   = eco-vm

CPU    = 1
MEMORY = 1024

OS = [ kernel     = /vmlinuz,
       initrd     = /initrd.img,
       root       = sda1,
       kernel_cmd = "ro xencons=tty console=tty1"]

DISK = [ IMAGE_ID = <%= erb_vm_info[:img_id] %> ]

NIC = [ NETWORK_ID = 4 ]

IMAGE_ID = <%= erb_vm_info[:ec2_img_id] %>
INSTANCE_TYPE = <%= erb_vm_info[:instance_type ]%>

<% if erb_vm_info[:user_data] %>
CONTEXT = [ 
	EC2_USER_DATA = "<%= erb_vm_info[:user_data] %>" ,
	TARGET = "hdc"
	]
<% end %>

* Add a ''VM_TYPE'' attribute to ''/etc/one/econe.conf'' with the NAME for the new type and the ''TEMPLATE'' that should be use:


# VM types allowed and its template file (inside templates directory)
:instance_types:
  :m1.small:
    :template: m1.small.erb
  :m1.large:
    :template: m1.large.erb

:!: The templates are processed by the EC2 server to include specific data for the instance, you should not need to modify the <%= ... %> compounds. Start by adjusting the OS, CPU and MEMORY to your needs.

====== Starting the Cloud Service ====== To start the EC2 Query service just issue the following command $ econe-server start You can find the econe server log file in ''/var/log/one/econe-server.log''. To stop the EC2 Query service: $ econe-server stop ====== Advanced Configuration ====== ===== Authorization methods ===== OpenNebula EC2 Server supports two authentication methods. The method can be set in the [[#econe-server.conf]], as explained above. These two methods are: ==== EC2 Auth ==== In the EC2 mode, a signatured is generated based on the user credentials. ==== x509 Auth ==== This method performs the login to OpenNebula based on a x509 certificate DN (Distinguished Name). The DN is extracted from the certificate and matched to the password value in the user database (remember, spaces are removed from DNs). The user password has to be changed running one of the following commands oneuser chauth new_user x509 "/C=ES/O=ONE/OU=DEV/CN=clouduser" oneuser chauth new_user --x509 --cert /tmp/my_cert.pem or create a new user: oneuser create new_user "/C=ES/O=ONE/OU=DEV/CN=clouduser" --driver x509 oneuser create new_user --x509 --cert /tmp/my_cert.pem To enable this login method, set the **'':auth:''** option of ''/etc/one/sunstone-server.conf'' to **''x509''**:


    :auth: x509

Note that OpenNebula will not verify that the user is holding a valid certificate at the time of login: this is expected to be done by the external container of the EC2 server (normally Apache), whose job is to tell the user's client that the site requires a user certificate and to check that the certificate is consistently signed by the chosen Certificate Authority (CA). Users with 'public' driver FIXME ===== Configuring a SSL Proxy ===== OpenNebula EC2 Query Service runs natively just on normal HTTP connections. If the extra security provided by SSL is needed, a proxy can be set up to handle the SSL connection that forwards the petition to the EC2 Query Service and takes back the answer to the client. This set up needs: * A server certificate for the SSL connections * An HTTP proxy that understands SSL * EC2Query Service configuration to accept petitions from the proxy If you want to try out the SSL setup easily, you can find in the following lines an example to set a self-signed certificate to be used by a lighttpd configured to act as an HTTP proxy to a correctly configured EC2 Query Service. Let's assume the server were the lighttpd proxy is going to be started is called ''cloudserver.org''. Therefore, the steps are: ==== 1. Snakeoil Server Certificate ==== We are going to generate a snakeoil certificate. If using an Ubuntu system follow the next steps (otherwise your milleage may vary, but not a lot): * Install the ''ssl-cert'' package $ sudo apt-get install ssl-cert * Generate the certificate $ sudo /usr/sbin/make-ssl-cert generate-default-snakeoil * As we are using lighttpd, we need to append the private key with the certificate to obtain a server certificate valid to lighttpd $ sudo cat /etc/ssl/private/ssl-cert-snakeoil.key /etc/ssl/certs/ssl-cert-snakeoil.pem > /etc/lighttpd/server.pem ==== 2. lighttpd as a SSL HTTP Proxy ==== You will need to edit the ''/etc/lighttpd/lighttpd.conf'' configuration file and * Add the following modules (if not present already) * mod_access * mod_alias * mod_proxy * mod_accesslog * mod_compress * Change the server port to 443 if you are going to run lighttpd as root, or any number above 1024 otherwise:


server.port               = 8443

* Add the proxy module section:


#### proxy module
## read proxy.txt for more info
proxy.server               = ( "" =>
                                ("" =>
                                 (
                                   "host" => "127.0.0.1",
                                   "port" => 4567
                                 )
                                 )
                             )


#### SSL engine
ssl.engine                 = "enable"
ssl.pemfile                = "/etc/lighttpd/server.pem"

The host must be the server hostname of the computer running the EC2Query Service, and the port the one that the EC2Query Service is running on. ==== 3. EC2Query Service Configuration ==== The ''econe.conf'' needs to define the following:


# Host and port where OCA server will run
SERVER=127.0.0.1
PORT=4567	
# SSL proxy that serves the API (set if is being used)
SSL_SERVER=cloudserver.org

Once the lighttpd server is started, EC2Query petitions using HTTPS uris can be directed to ''https://cloudserver.org:8443'', that will then be unencrypted, passed to localhost, port 4567, satisfied (hopefully), encrypted again and then passed back to the client. ===== Using an specific group for EC2 ===== It is recommended to create a new group to handle the ec2 cloud users: $ onegroup create ec2 ID: 100 Create and add the users to the ec2 group (ID:100): $ oneuser create clouduser my_password ID: 12 $ oneuser chgrp 12 100 Also, you will have to create ACL rules so that the cloud users are able to deploy their VMs in the allowed hosts. $ onehost list ID NAME RVM TCPU FCPU ACPU TMEM FMEM AMEM STAT 0 kvm01 0 800 800 800 7.8G 6.7G 7.8G on 1 xen01 0 800 800 800 7.8G 6.7G 7.8G on 3 kvm03 0 100 99 100 2G 512M 2G on These rules will allow users inside the ec2 group (ID:100) to deploy VMs in the hosts kvm01 (ID:0) and kvm03 (ID:3) $ oneacl create "@100 HOST/#1 MANAGE" $ oneacl create "@100 HOST/#3 MANAGE" You **have to create a VNet network** using the ''onevnet utility'' with the IP's you want to lease to the VMs created with the EC2 Query service. $ onevnet create /tmp/templates/vnet ID: 12 Remember that you will have to add this VNet (ID:12) to the users group (ID:100) and give USE (640) permissions to the group in order to get leases from it. $ onevnet chgrp 12 100 $ onevnet chmod 12 640

:!: You will have to update the NIC template, inside the ''/etc/one/ec2query_templates'' directory, in order to use this VNet ID

===== Enabling Elastic IP functionality ===== An Elastic IP address is associated with the user, not a particular instance, and the user controls that address until he chooses to release it. This way the user can programmatically remap his public IP addresses to any of his instances. In order to enable this functionality you have to follow the following steps: ==== 1. Create a VNET containing the Elastic IPS ==== * As oneadmin create a new FIXED VirtualNetwork containing the public IPs that will be controlled by the EC2 users:


NAME    = "ElasticIPs"
TYPE    = FIXED

PHYDEV  = "eth0"
VLAN    = "YES"
VLAN_ID = 50
BRIDGE  = "brhm"
 
LEASES  = [IP=10.0.0.1]
LEASES  = [IP=10.0.0.2]
LEASES  = [IP=10.0.0.3]
LEASES  = [IP=10.0.0.4]
 
# Custom Attributes to be used in Context
GATEWAY = 130.10.0.1

$ onevnet create /tmp/fixed.vnet ID: 8

This VNET will be managed by the oneadmin user, therefore ''USE'' permission for the ec2 users is not required

* Update the econe.conf file with the VNET ID:


:elastic_ips_vnet: 8

* Provide associate and disassociate scripts The interaction with the infrastructure has been abstracted, therefore two scripts have to be provided by the cloud administrator in order to interact with each specific network configuration. This two scripts enable us to adapt this feature to different configurations and data centers. These scripts are language agnostic and their path has to be specified in the econe configuration file:


  :associate_script: /usr/bin/associate_ip.sh
  :disassociate_script: /usr/bin/disassociate_ip.sh

The associate script will receivd three arguments: **elastic_ip** to be associated; **private_ip** of the instance; **Virtual Network template** base64 encoded The disassociate script will receive three arguments: **elastic_ip** to be disassociated

Scripts to interact with OpenFlow can be found in the following ecosystem project FIXME