FOSS Repository

What it Takes To Be An OpenSource Developer - Πώς γίνεσαι OpenSource developer;

Spring Web Flow (Rich Client Java Web Framework)

15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
Open source Google-style large scale 
data analysis with Hadoop
Email: ikons@cslab.ece.ntua.gr
Web: http://www.cslab.ntua.gr/~ikons
Computing Systems Laboratory
School of Electrical and Computer Engineering
National Technical University of Athens
Ioannis Konstantinou 
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
Big Data
• Facebook: 20TB/day compressed
• CERN/LHC: 40TB/day (15PB/year)
• NYSE: 1TB/day
• 2009 Digital Universe: 800.000 Petabytes or 
0.8 Zettabytes
• Moore's Law: Data doubles every 18 months
• 2020 prediction: 35 Zettabytes (44 times 
bigger than 2009)
2/18
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
What is Hadoop?
• It’s a distributed framework for large-scale 
data processing:
• Inspired by Google’s architecture: Map 
Reduce and Google File System
• Can scale to thousands of nodes and 
petabytes of data
• A top-level Apache project (since 2008) –
Hadoop is open source
• Written in Java, plus a few shell scripts
3/18
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
Why Hadoop?
• Hadoop is designed to run on cheap 
commodity hardware
• Fault-tolerant hardware is expensive
• It automatically handles data replication and 
node failure
• It does the hard work – you can focus on 
processing data
4/18
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
When to use Hadoop?
• There is access to lots of commodity hardware
• The processing can be easily parallelized
• Need to process lots of unstructured data
– Data intensive applications
• It is ok to run batch jobs (no need for 
interactive results)
5/18
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
Architecture
• HDFS: Distributed file system
– Hard to store a PB
– Based on Google FS
– Fault-tolerant: handles replication, node failure, etc
• MapReduce : Data aware parallel computation 
framework
– Even harder to process a PB
– Based on a research paper by Google
6/18
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
Hadoop Distributed File System 1/2
• Master/Slave Architecture
• Files are split into one or more blocks and 
these blocks are stored in a set of DataNodes
• A Master NameNode 
– a master server that manages the file system 
namespace and regulates access to files by clients
– determines the mapping of blocks to DataNodes 
• Many DataNodes
– Serve client read/write requests
– Create/delete/replicate blocks 
7/18
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
Hadoop Distributed File System 2/2
8/18
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
MapReduce 1/3
• A programming model 
• A software framework 
• for writing applications that 
– rapidly process vast amounts of data in parallel 
– on large clusters of compute nodes
9/18
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
MapReduce 2/3
• Problem is separated in two different phases, 
the Map and Reduce phase.
• Map: Non overlapping chunks of input data is 
assigned to separate processes (mappers) that 
emit a set of intermediate results
• Reduce: Map results are fed to a usually 
smaller number of  processes called reducers 
that “summarize” their input in a smaller 
number of results
10/18
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
MapReduce 3/3
11/18
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
When should I use it?
• Good choice for
– Indexing log files
– Sorting vast amounts of data
– Image analysis
• Bad choice for
– Figuring π to 1,000,000 digits
– Calculating Fibonacci sequences
– MySQL replacement
12/18
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
Hadoop MapReduce
• Master/Slave architecture
• A JobTracker Master
– Runs together with NameNode
– Receives client job requests
– Schedules and monitors MR jobs 
• Move computation near the data
• Speculative execution
• Many TaskTrackers
– Run together with DataNodes
– Perform I/O operations with DataNodes
13/18
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
Typical problems
• Log and/or clickstream analysis of various kinds
• Marketing analytics
• Machine learning and/or sophisticated data 
mining
• Image processing
• Processing of XML messages
• Web crawling and/or text processing
• General archiving, including of relational/tabular 
data, e.g. for compliance
14/18
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
Use cases 1/3
• Large Scale Image Conversions
• 100 Amazon EC2 Instances, 4TB raw TIFF data
• 11 Million PDF in 24 hours and 240$
• Internal log processing
• Reporting, analytics and machine learning
• Cluster of 1110 machines, 8800 cores and 
12PB raw storage
• Open source contributors (Hive)
• Store and process tweets, logs, etc
• Open source contributors (hadoop-lzo)
15/18
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
Use cases 2/3
• 100.000 CPUs in 25.000 computers
• Content/Ads Optimization, Search index
• Machine learning (e.g. spam filtering)
• Open source contributors (Pig)
• Natural language search (through Powerset)
• 400 nodes in EC2, storage in S3
• Open source contributors (!) to HBase
• ElasticMapReduce service
• On demand elastic Hadoop clusters for the 
Cloud
16/18
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
Use cases 3/3
• ETL processing, statistics generation
• Advanced algorithms for behavioral analysis 
and targeting
• Used for discovering People you May Know, 
and for other apps
• 3X30 node cluster, 16GB RAM and 8TB storage
• Leading Chinese language search engine
• Search log analysis, data mining
• 300TB per week
• 10 to 500 node clusters
17/18
15/5/2010 ΕΛ/ΛΑΚ Conference,  Athens Ioannis Konstantinou
Questions
18/18


Open source Google-style large scale data analysis with Hadoop

Το εργαλείο επεξεργασίας αριθμητικών δεδομένων Octave

BOINC CS, Κατανεμημένα συστήματα με μαθητικούς φορητούς OLPC XO

Ανάπτυξη Εφαρμογής για το OLPC

Αυτοματοποιημένο Σύστημα Αξιολόγησης αξιοποιώντας το λογισμικό TCExam

Η χρήση του Ελεύθερου Λογισμικού/Λογισμικού Ανοικτού Κώδικα (ΕΛ/ΛΑΚ) στη δημόσια Δευτεροβάθμια Εκπαίδευση: Αξιολογώντας τις στάσεις και τις απόψεις των Εκπαιδευτικών Πληροφορικής

Open Source & the
Public Sector
the JBoss/Red Hat experience
Dimitris Andreadis
Software Engineering Manager
JBoss Application Server
JBoss, by Red Hat
11th Greek ICT Forum
November 4-5th, 2009, Athens
Agenda
 What is JBoss
 Public Sector JBoss Use Cases
 Open Source – A Strategic Choice
What is JBoss?
 An Open Source Java EE* Application Server
 Founded by Marc Fleury in 1999
 Backed by an international community of developers
 Developer’s #1 Choice
 Innovative Design
 Easy to Use
 LGPL Licensed
 Freely Use/Distribute/Embed
 Closed sourced Servers at $10k - $100k+
JBoss – the project
*Enterprise Edition
OpenSource + Recession = Love!
.com bust
subprimes
…
Red Hat acquisition of JBoss (2006)
 Red Hat offered $350M for JBoss Inc.
 JBoss formed the Middleware division of Red Hat
 An Open Source stack
for the Enterprise, at a
fraction of the cost
JBoss Today
 50+ Community Projects
 Multiple product lines
 25+ Million Downloads
 500,000+ Users
 700+ Contributors
 ~30% of World’s Java 
Enterprise Applications
use JBoss Technologies
Magic Quadrant for Enterprise Application Servers 
(Source: Gartner April 2008)
Public Sector
JBoss Use Cases
Who is using JBoss?
(tip of iceberg)
Financial Travel Gov Media/
Entertainment
InsuranceTelecom
Some Public Sector References
• Spanish Ministry of Education and Science
• CSI-Piemonte, modernizing Italian local public administration
• SRI Ecuador, Online Services to Taxpayers
• Brazilian National Institute of Educational Research and 
Studies
• Brazilian Federal Supreme Tribunal’s Technology Office
• London Underground (Oyster Card)
• US Army, Navy, Air Force, NSA.
• Swiss Post Office
• …
French Tax Agency – Project Copernicus
 Started in 2004 with the goal to restructure
the French equivalent of Taxis NET
 1 Billion Euros budget over 10 years
 JBoss on top of 250 RHEL servers, 80Tb of Data
 JBoss chosen on technical merits among 2 FOSS and 3 
proprietary servers
 TCO savings 75% (13M Euros) for the application server part
 Handles all Tax declarations & payments since 2006
NASA - Spacecraft  Trajectory Operations
 Replaces legacy systems with JBoss Middleware for all of 
the Trajectory Analysis, Pre-mission Design, Operations, and 
Post-Flight Assessments for the Human Space Flight 
Program
− Space Shuttle Program (SSP)
− International Space Station (ISS)
− Constellation Program (CxP) and other vehicles
Open Source
A Strategic Choice
 Lack of standards
 Closed formats
 Islands of information
 No Interoperability
 No extensibility points
 …
Common Themes:
 Replace a failed system with 
another (failed system)
 Vendor Lock-in
 License costs
Open Source Key Benefits
 Promotes Transfer of Knowledge
 Endorses Standards Adoption
 Open Data Formats & Interfaces
 Avoids Vendor Lock-In
 Enhances Security
 Reduces TCO
 …
 Software As A Service (SaaS)
Open Source Adoption Problems
 Lack of Know-How
 Decision makers do not understand Open Source
 Often considered Free (as in Beer) or Immature
 Competitive Market
 Do customers understand their needs?
 Do customers choose the most competitive offer?
 Chicken/Egg Problems
 Cannot choose Open Source if there is no support
 Cannot support Open Source due to little demand
and Solutions
 Academia
 Open Source Curriculum
 Hands-on training
 Companies
 Invest in People
 Technical Careers
 Government
 Enforce Transparency
 Establish Open Source
Usage Guidelines
dimitris@redhat.com http://www.jboss.org


Open Source & the Public Sector: the JBoss/Red Hat experience

Olivier Beutels 
Sales Manager SkySQL Ab 
 
Vangelis Katsikaros  
Partner 
Products and 
Solutions 
1101.02.02 Oracle, MySQL and InnoDB are registered trademarks of Oracle and/or its affiliates.  Other names may be trademarks of their respective 
owners. 
2 
Agenda 
• Intro to SkySQL 
• SkySQL Enterprise 
• SkySQL Monitoring Tools 
• SkySQL Services 
 
3 
Intro to 
SkySQL 
4 
The MySQL AB Heritage 
• MySQL® AB was: 
–70K+ downloads per day since 2003 
–13M+ active installations worldwide 
–The largest OSS exit to-date with $1B purchase by 
Sun 
 
• MySQL® is: 
–Part of Oracle via the acquisition of Sun 
Microsystems 
–Developed and improved by Oracle due to a 
controversial commitment with the EU 
5 
16 of the 20 most popular websites use 
MySQL as main front-end Database 
 
 
1.Google 
2.Facebook 
3.YouTube 
4.Yahoo! 
5.Windows Live 
6.Baidu 
7.Blogger 
8.Wikipedia 
9.QQ.com 
10.Twitter 
11.MSN 
12.Yahoo! JP 
13.Amazon.com 
14.Taobao 
15.Google IN 
16.Sina 
17.Google DE 
18.Google HK 
19.Wordpress 
20.eBay 
Source: Alexa Top Sites - 9 January 2011 
http://www.alexa.com/topsites 
6 
SkySQL Ab 
• Funded by: 
–MySQL AB founders Monty Widenius and David 
Axmark 
–US Investment group OnCorps.org 
• Operating in 13 countries, with 90% of the 
team ex-MySQL® AB employees 
• Backed by: 
–Product Engineering MontyProgram Ab 
–Top Community contributors and End Users 
7 
SkySQL Goals 
• To be #1 worldwide provider of MySQL®-
related offerings 
• To build a profitable, sustainable commercial 
business while preserving open source 
values 
• To raise the bar on the quality of MySQL® 
service and support 
• To ensure and promote competition 
8 
SkySQL Offering 
• SkySQL Enterprise Subscriptions 
–Monitoring, Administration and End User tools 
–L0 to L3 Technical and Consultative Support on the 
most famous and used MySQL® distributions and 
forks 
• SkySQL Consulting 
–Top class team for MySQL® technology 
–Extended service offering from Health Check to 
continuous administration 
• SkySQL Training 
–MySQL® Training and Certification 
9 
SkySQL Customers 
“…We highly recommend SkySQL for 
their ability to quickly understand the 
domain they need to operate in and for 
their expertise in MySQL® Cluster.” 
“We chose to renew with SkySQL because it 
could offer us a more cost-effective, yet still highly 
competitive enterprise-class software and support 
solution for our MySQL® databases…SkySQL 
offers MySQL® customers like Easyflirt.com a 
highly compelling alternative to Oracle software, 
service, and support.” 
Deutsche Telekom AG  
• Deutsche Telekom AG is one of the world's 
leading telecommunications and information 
technology service companies. With almost 200 
million customers, DTAG is present in over 50 
countries. 
• SkySQL is the 24/7 support provider for DTAG's 
1000+ MySQL installations in EMEA. 
• http://www.skysql.com/en/customers/featured-
customers 
 
10 
11 
SkySQL Enterprise Offering 
Server MariaDB Server 
• L0 to L3 support 
• Hot Fixes and Patches 
• Monitoring Tools 
• User and DBA Tools 
• Field Services 
Enterprise 
12 
SkySQL 
Enterprise 
13 
• What it is: 
– A comprehensive offering of server software, 
production support and monitoring tools for any 
version of the MySQL® database 
• What it includes: 
– MySQL® Server 
– MariaDB Server 
– SkySQL Enterprise Monitor 
– SkySQL Visual Editor 
– SkySQL Technical Support 
Enterprise 
14 
SkySQL Enterprise Features 
Basic Silver Platinum Unlimited 
Included Software 
MySQL® ✔ ✔ ✔ ✔ 
MariaDB ✔ ✔ ✔ ✔ 
High Availability ✘ ✘ ✔1 ✔1 
Included Tools 
Visual Query Editor with Graphical Query Builder ✘ ✔ ✔ ✔ 
Enterprise Monitor with Real time Query Analyzer ✘ ✘ ✔ ✔ 
Problem Resolution Support 
Incidents 2 Unlimited Unlimited Unlimited 
Web Based Support Portal ✔ ✔ ✔ ✔ 
Emergency Phone Access2 ✘ Business Hrs 24x7 24x7 
Emergency Response Time 30 min 30 min 
Hot Fixes ✘ ✔ ✔ ✔ 
Consultative Support 
Code Review ✘ ✘ ✔ ✔ 
Login Support ✘ ✘ ✔ ✔ 
Performance Tuning ✘ ✘ ✔ ✔ 
On-Site Health Check ✘ ✘ ✔1 ✔ 
1: Option - 2: Callback 
15 
at a glance 
 
• MySQL® GPL versions 3.23, 
4.X and 5.X 
• The MySQL® storage engines 
available in the distribution 
(MySQL®/NDB available 
through the Cluster Practice) 
• Supported platforms: 
Linux, Windows and Solaris 
Enterprise 
16 
at a glance 
 
• MariaDB 5.1 & 5.2 
• The MariaDB storage engines 
available in the distribution 
• Supported platforms: 
Linux, Windows and Solaris 
Enterprise 
17 
at a glance 
• MySQL® Replication 
• Linbit DRBD with Heartbeat 
 
• Linux and Windows OS 
Clustering with shared 
storage 
• MySQL® Cluster/NDB 
• Virtualised Systems 
Note: MySQL Replication and Linbit DRBD are fully supported. SkySQL Enterprise and NDB can be installed 
and configured on Linux and Windows Clustered Systems with the help of SkySQL Professional Services. 
Virtualised systems are supported, provided the guest OS is a supported SkySQL Enterprise Platform. 
Enterprise 
18 
at a glance 
 
• Query Profiler 
• Index Analyser 
• Schema Optimiser 
• HTTP and SSH Tunnelling 
connections 
• Visual Query Builder 
• Visual Schema Designer 
• Data and Schema Synchronisation 
Enterprise 
19 
at a glance 
 
• Agent-less, Cloud-ready with SSH 
Tunnelling 
• Real-time Query and Log Analyser 
• Replication Monitor 
• Deadlock Monitor 
• Resources Monitor (CPU, Memory, 
Disks) 
• Historical Analysis and Profiling 
• SNMP Traps 
Enterprise 
20 
SkySQL Enterprise vs Oracle MySQL® Ent. 
SkySQL Enterprise  Oracle MySQL Enterprise 
Supported distribution Oracle MySQL®, MariaDB Oracle MySQL® 
Licensing GPL Commercial 
Supported versions with no 
extra cost 
3.23, 4.X, 5.X 5.1, 5.5 
Available engines 
MyISAM, Aria, InnoDB, XtraDB, PBXT, FederatedX, 
Archive, Memory, CSV, OQGraph, SphinxSE, TokuDB 
MyISAM, InnoDB, Archive, 
Memory, CSV 
Supported High Availability 
platforms 
MySQL® Replication, DRBD MySQL® Replication 
High Availability Platforms 
Through PS Practice 
Linux and Windows OS Clustering, 
Cluster/NDB 
- 
Monitoring tools 
Agent-less SkySQL Enterprise Monitor with 
Log and Query Analyzer 
MySQL® Enterprise Monitor with 
Query Analyzer MySQL® 
Workbench with Data Modeller 
User tools 
SkySQL Visual Query Editor with Data 
Modeller and Query Builder 
MySQL® Workbench with 
Data Modeller 
VM support Linux, Windows & Solaris Guest OSs - 
Price per Server 
 
€3,000 €4,000 1  
Extra features from SkySQL 
1 Up to 4 sockets 
21 
Support1 
• L0 to L2 Support provided by the SkySQL team: 
–Talk with MySQL experts since L0 
–24x7 access with 30 min response time 
–Web, Email and Phone2 Access 
–L3 Support provided by Monty Program Ab 
• Hot fixes for the supported platforms 
• Patch submit follow up 
• Consultative Support provided by the SkySQL 
team: 
–Code Review, Login Support, Performance and Tuning 
–On-Site Health Check as an Option for Platinum, 
standard for Unlimited. 
1 Features depend on the subscription level. Platinum and Unlimited include all the features presented in this slide 
2 Call-back 
Enterprise 
22 
SkySQL 
Monitoring Tools 
23 
• A "MySQL® DBA in a box'' that helps MySQL® DBAs 
manage more MySQL servers, tune their current 
MySQL® servers, resolve query bottlenecks and fix 
problems with their MySQL® database applications, 
before they can become serious problems or costly 
outages.  
 
• Agent-less, cloud-ready solution 
–Active through HTTP and SSH 
• Enterprise-rich features: 
–MySQL® Monitor and Advisors 
–Systems Monitoring 
–Log Analysis 
–Query Analysis 
–Change Tracking 
–Javascript Extensions 
Monitor Enterprise 
24 
Monitor Features Enterprise 
Multi-User Management 
Filter access to specific servers for individual users 
Specify different permissions for different users 
Find Problem SQL by 
Querying MySQL® Proxy 
Analysing General Query Log 
Analysing Slow Query Log 
Monitoring queries in real-time 
CSV exports 
Security 
Detect MySQL® hacking attempt 
Identify and fix security vulnerabilities 
Monitor Replication 
Alerts on replication delays 
Replication log details 
Server Optimization 
Deadlock Monitoring 
Server Disk Usage 
History/Trend Analysis 
Performance and tuning optimization 
Error Log Monitoring 
Multi-Platform Support 
Linux and Windows 
32 and 64 bits 
Full and silent installers 
HTTP and SSH Tunnelling 
Miscellaneous features 
Side-by-side comparisons 
AJAX interface 
Built-in web server 
Customisable Javascript and MONyog Object Model 
Custom advisors, alerts and charts 
Email alerts 
SNMP Traps 
25 
SkySQL Enterprise Monitor and 
MySQL® Enterprise Monitor 
Feature SkySQL Enterprise Monitor MySQL Enterprise Monitor 
DB Server Agent 
Agent-less: the tool can be used in public or private 
clouds without installing additional software on the 
DB Server 
Agent-based: an agent must be installed on the DB 
Server. 
Tracking 
Track changes: changes can be tracked to analyse 
possible issues related to the change 
Cannot track changes 
Data Export 
Export to Excel: reports or missing graphs can be 
created in Excel or OpenOffice 
No export capabilities 
Process List 
Online PROCESSLIST: Threads can be monitored 
in realtime and they may be stopped 
No PROCESSLIST  
Logs Online Slow, General and Error Log analysis No Logging analysis 
Query Analyzer General Log or Query Sniffer using MySQL Proxy 
Query Analyzer using MySQL Proxy or Commercial 
Connectors 
Sampling 
Customised granularity to 1 second. Sampling 
selection through the user interface 
No sampling selection. Customised granularity to 1 
minute. 
Graphs Export Export to multiple graphs format No export capabilities 
Disk Info and Usage Disk info by Schema and Table No Disk Info Capabilities 
Server view 
Single or Comparative view: it can be used to 
analyse the effectiveness of balanced services 
Single or combined: it can be used to analyse the 
overall throughput of the DB infrastructure 
Timeframe selection 
Date/Time User interface: the user must select a 
specific interval 
Point and click selection: the user can highlight a 
timeframe to analyse the queries during that period 
26 
Monitor - Advisors Enterprise 
27 
Monitor - Dashboard Enterprise 
28 
Monitor - Disk Usage Enterprise 
29 
Monitor - Query 
Analyzer 
Enterprise 
30 
Monitor - Server 
Configuration 
Enterprise 
31 
SkySQL 
Services 
32 
Professional Services Packages 
• Time Hire 
• Database Healthcare 
• Performance Tuning 
• Architecture and Design 
• High Availability 
• Cluster Practice 
• Database Migration 
• Customized Consulting 
33 
• Consulting evaluation and assessment 
–Technical qualification 
– Initial assessment and proposal 
• Service Delivery 
–Onsite or remote 
–Flexible scheduling 
–Best Practices provided by the most skilled MySQL® team 
WW 
–Technical Account Management and Project Leading on 
request 
• Engagement 
–Engagement Summaries and Reports 
–Time Sheet approval 
–Quality Assurance surveys 
–Business follow ups and continuity 
Professional Services 
General Guidelines 
40 
SkySQL Training 
• MySQL® Administration 
• MySQL® Performance / Tuning 
• MySQL® Development 
• MySQL® High Availability 
• MySQL Cluster® 
42 
Thank You! 
Olivier Beutels 
Sales Manager 
olivier@skysql.com 
 
Vangelis Katsikaros  
Partner 
 


Not a member yet