Showing posts with label Big Data. Show all posts
Showing posts with label Big Data. Show all posts

Tuesday, November 3, 2015

50 diagrams illustrating IoT to Cloud/Big Data Analytics interfaces



My colleague Bob Marcus  collected about 50 diagrams (with reference URLs) showing alternative architectural approaches to interfacing IoT to Cloud/Big Data Analytics. See the Link below. The diagrams are generally very high level with more details available on the source Web sites.  



50 diagrams illustrating IoT to Cloud/Big Data Analytics interfaces

Sunday, July 26, 2015

Installing Vertica Analytics and Management Console




In this blog we will  learn how to install vertica analytic  and management console.

Before we install vertica, we need to understand the cluster configuration for vertica. each node in cluster has unique ip address, they dont share common storage.
Each hosts has a file /etc/hosts so that hosts can identify each other.




there are also dedicated ports that need to be left opened for vertica communications like 5434, 5444,5450...

besides root user, 2 more users need to be created.:
1) dbadmin: for vertica database
2) mcadmin for management console

root users install vertica analytics and management console on each host in the cluster.

on of the host create a directory,to hold vertica database and manaegment console installation files.

On all of the host create a directory for vertica database.

Vertica support following unix OS.
'


To install vertica, you just need to unzip vertica rpm file . Once that is done, it will create a directory /opt/vertica.

The vertica directory contains many sub directory




Now we need to install vertica


here node is the ip address of the different machines. t will install vertica on all hosts. Now hosts become nodes.

for licensing vertica, we now need to login as dbadmin. run the command admintools.  a pop will be there for you to define full path of licensing file



Now we will install management console.

run the installation command

Now open the browser and type the URL, https://node1:5450   here node1 is the nodename.  accept the license agreement

create administrative user for management console.






and its done.

Tuesday, July 21, 2015

HP Vertica Building Blocks





HP vertica the physical architecture is designed to distribute physical storage and to allow parallel query execution over a potentially large collection of computing resources.
The most important terms to understand are instance, node, cluster, and data:
NODE: a node is a machine configure to run vertica
Instance: Its the vertica instance installed on the node
Cluster: A cluster consists of collection of nodes
data: Data is the data in the vertica database distributed across the nodes

Vertica platform is build ground up. there are 6 important vertica features:


Columnar Orientation: Data is stored in columns instead of rows, thus saving memory and I /O
Advance Compression:Operates on compressed and encoded data delivers superior performance on a smaller storage footprint    and reduced capital expense

High availability: Vertica is designed for high availability. to achieve this vertica uses projections or copy of projections stored in different nodes. number of projections are based on  K-Safety=1 
K-Safety is the measure of fault tolerance of the database. it support values 0,1,2. inorder to enable K-Safety there should be minimum of 3 nodes in the cluster

Automatic Database design: The database designer allows you to optimize, tune and control your database without shutting down the database

Application integration: Using standard SQL, there is seamless integration with robust and ever growing analytical solutions

Massively Parallel Processing:automatic,limitless, linear scaling of peer to peer nodes, each processing an allocated subset of data


Sunday, August 31, 2014

Cloud Computing and Big Data: Threat to IT Jobs and Over Hyped?

There are two disruptive technologies which have gained significant traction in the world economy: Cloud Computing and Big Data.

Lets discuss some opportunities and threats and realistic scenarios regarding these technologies.

Cloud Computing:

The world is seeing significant transformation in business with the arrival of cloud. In this blog, our focus is on public cloud.

More and More large and small enterprises are moving to public cloud hosted by Amazon, Microsoft and HP. Software as a Service (SAAS) is the hottest service available. All small and big software vendors have started hosting their software on the public and offer as a service. This frees up organizations headache in purchasing hardware and software.
Pay as you go makes lots of sense to an organization. There is faster turnover  to transition to service in cloud then in your own Data centers.

Since multiple vendors are hosting their software on cloud and offering it as a service, they are also taking ownership of availability of service and management of service. This affects IT jobs in Operation. As more and more servers are built on cloud, the less jobs remain for operations team. This results in lots of jobs to me lost. This reduces the cost of an enterprises. So the managed services model of the IT companies is dying. Dont be surprised if managed service model is completely dead in 5 years.

Now there is an interesting issue here. The challenge for an enterprises will be multiple vendor management. There will be a situation when there will too many software vendors to be managed so that  enterprise starts bringing the cloud back to its Data center. Its a possibility. Circle of life. Lets wait and watch


Big Data

Let me start with the statement that Big Data technologies are awesome and its potential for enterprise is enormous.
But when we talk about business opportunities in big data solution from IT perspective, i think its over hyped.

Let em explain you why.

Its an awesome opportunity if you have to stand up multi node clusters for big data. But this is not true anymore thanks to Amazon and Microsoft Azure. I dont need to set up expensive cluster to get big data solution for my enterprise. I can just go to Amazon EMR or Microsoft Azure HDinsight and spin up and instance and get my big data solution in few hundred dollars.

SO for IT companies Big Data wont make much money too.





Friday, August 22, 2014

Big Data: Microsoft HDinsight


So far we have covered following topics in Big Data


In this blog we will discuss  Microsoft HDinsight.

HD Insight Overview

- Microsoft Hadoop distribution
-local and cloud hadoop support
-Local: HDinsight server on HDP( hortonworks data platform)
-Cloud:HDinsight on Azure service


HDinsight advantages:
-Full integration with MS BI stack
-SQL server(RDBMS,SSAS,SSRS)
-Sharepoint (Powerview)
-office (Excel)

Lets go to windows azure

here you can create a free trial account. I have already one account

HDinsight is not fully available in windows azure. to enable it you need to go to PREVIEW FEATURES

and enable it.


Thats it. all you need to go to dashboard.

Thursday, August 21, 2014

Big Data: AMAZON ELASTIC MAPREDUCE (EMR)


So far we have covered following topics in Big Data

                         Big Data- The Rise and the Future

·         Big data: Technology Stack

·         Big Data: Hadoop Distributed Filesystem (HDFS)

·         Big Data: Map Reduce

·         Big Data- Installing Hadoop ( Single Node)

·         Big Data- Apache Hadoop Multi Node

·         Big Data: Troubleshooting, Administering and optimizing Hadoop

·         Big Data: Managing HDFS

·         Big Data: Map Reduce Development

·         Big Data: Introduction to Pig


In this blog we will discuss AMAZON ELASTIC MAPREDUCE (EMR)

AMAZON ELASTIC MAPREDUCE (EMR) Overview

What is EMR?
-Webservice on top of AWS that uses EC2 for processing and S3 for storage
-Data is pulled from S3, processed by auto-configured EC2 cluster and results pushed back to S3
-Crunch your data in the cloud without the hassle of managing your own cluster/infrastructure!!















What is an EMR Job Flow?
-Data processing wizard
-Hive,mapreduce, hbase and pig


The only thing we need to do is configure EMR Job Flow. Once its configured, rest is very easy. Even EMR JOB FLOW is very easy in amazon.

Thats it.


Saturday, August 16, 2014

Big Data: Introduction to Zookeeper

So far we have covered following topics in Big Data

                         Big Data- The Rise and the Future

·         Big data: Technology Stack

·         Big Data: Hadoop Distributed Filesystem (HDFS)

·         Big Data: Map Reduce

·         Big Data- Installing Hadoop ( Single Node)

·         Big Data- Apache Hadoop Multi Node

·         Big Data: Troubleshooting, Administering and optimizing Hadoop

·         Big Data: Managing HDFS

·         Big Data: Map Reduce Development

·         Big Data: Introduction to Pig


In this blog we will discuss ZooKeeper.

In the world of hadoop, theme is distributed. What if you want to build your own distributed application?
You have to worry about centralized configuration, synchronization, serialization.

Zookeeper is the distributed coordination service for the distributed application. a centralized repository.

ZOOKEEPER Overview

What is zookeeper?
-Distributed coordination service for distributed applications
-Used for synchronization, serialization and coordination
-Handles the 'nitty-gritty' side of distributed app dev
-Apps use these services to coordinate distributed processing



Distributed Challenges:
- coordination is error prone
-Rack conditions, deadlocks,partial failures, inconsistencies

ZooKeeper Goals
-Serialization
-reliability
-Simple API
-Atomicity


Typical Uses
-Configuration - message queue
-Notification/Synchronization

Now Lets talk about ZOOKeeper architecture


Above diagram, if we looks closely
Z nodes
- Container for data and other nodes
-Stores Stats' user data( 1MB)

Z nodes types
- persistent
-emphermeral
-sequential


Thursday, August 14, 2014

Big Data: Introduction to Hbase

So far we have covered following topics in Big Data
                         Big Data- The Rise and the Future

·         Big data: Technology Stack

·         Big Data: Hadoop Distributed Filesystem (HDFS)

·         Big Data: Map Reduce

·         Big Data- Installing Hadoop ( Single Node)

·         Big Data- Apache Hadoop Multi Node

·         Big Data: Troubleshooting, Administering and optimizing Hadoop

·         Big Data: Managing HDFS

·         Big Data: Map Reduce Development

·         Big Data: Introduction to Pig


In this blog we will discuss Hbase.

HBASE OVERVIEW
What is HBase? It is
-Distributed Column Oriented Database on tap of Hadoop/HDFS: Those coming from relational database understand that there are Row oriented Database and column Oriented database. Hbase is column oriented database.

Row                    vs                   Column
-OLTP                                   -OLAP
-Single row                            -Aggregration but many rows and cloumns
Read/Write
-Smaller number of                 -High compression rates due to few distinct values
columns and rows
lets talk little high level, Hbase                    vs                RDBMS
-Schema less                                                           -Schema
-Wide                                                                      - Thin
-Denormalised                                                         - Normalised

How about differences between Hbase and HDFS?
HBase-Low latency access to single rows from billions of records
HDFS-High latency batch processing. No concept of random read/writes

HBASE ARCHITECTURE


We have another Master slave architecture here.So in this we have our master server and Region servers. Region servers are the slaves.Master server is compared to name node in HDFS.its gona manage and monitor Hbase cluster operations, assign region to region servers and do load balancing and spitting.

Zookeeper also plays major role in HBase, which we will discuss later. If you go to region server and go in detail



at the very top level, we see table, under table then we see region. Underneath the region is store.In the Store, there are memstore and store file. Memstore is the place where data comes in and later flushed to store file.

so this describes the Hbase architecture

Wednesday, August 13, 2014

MY TOP 50 Blogs

Hi,
I have written about 50 blogs. When you write so many blogs, articles gets lost. Below is the link of my all blogs, in one place

Cloud Computing

  1. vCoud Director Installation and configuration
  2. Changing Face of Managed services: A Threat to Likes of HP, IBM
  3. HP CSA Implementation
  4. Are HP and Intel About to Revolutionize Virtual Desktops? (HPQ, INTC)
  5. HP CSA Vs VMware VCAC
  6. 10 best practices for cloud design
  7. vCloud Suite Architecture-1
  8. vCloud Suite Architecture-2
  9. Application Suitability For Cloud
  10. Openstack- Its importance in Cloud. The HP Helion Boost


Big Data:

·         Big data: Technology Stack

·         Big Data: Hadoop Distributed Filesystem (HDFS)

·         Big Data: Map Reduce

·         Big Data- Installing Hadoop ( Single Node)

·         Big Data- Apache Hadoop Multi Node

·         Big Data: Troubleshooting, Administering and optimizing Hadoop

·         Big Data: Managing HDFS

·         Big Data: Map Reduce Development

·         Big Data: Introduction to Pig


Datacenter, Transformation and Migration

·         Data Center Migration

·         Application Migration Strategy

·         Green Datacenter


Enterprise architecture- Solution Architecture

·         TOGAF: Practical implementation-1

·         TOGAF: Practical Implementation-2: Requirement Gathering- Business Architecture

·         TOGAF: Practical Implementation-3: Requirement Gathering-Information Systems Architecture, Technology architecture and Opportunities=> Solution

 

 

 

Citrix:

Architecting a Citrix Virtualization solution
 There are 2 parts to it:
  1)    Assessment
  2)    Design

Assessment is further divided into
·          User Community
·         Operating System Delievery
·         Application Delivery
·         Server Virtualization
·         Infrastructure
·         Security and personalization
·          Operation and Support
·         Conceptual Architecture

Design is further divided into
·         Application Delievery design
·         Desktop Delivery design
·         Access Design

Featured Post

Ichimoku cloud

Here how you read a ichimoku cloud 1) Blue Converse line: It measures short term trend. it also shows minor support or resistance. Its ve...