Friday, April 14, 2017

loopback adapter on windows 10

How to install Microsoft Loopback Adapter on Windows 10To install the microsoft loopback adapter on win 10 you must:

  1. right click on window start menu icon and select Device manager. Device manager window will immediatel open (or you may use any other way how to open device manager window)
  2. click on Action, and select Add legacy hardware
  3. click Next on welcome screen
  4. choose "Install the hardware that i manually select from a list" and click on Next
  5. scroll down and select Network adapters from offered common hardware types and click on Next
  6. select Microsoft as the manufacturer, and then select Microsoft KM-TEST Loopback adapter card model, click on Next
  7. click on Next
  8. click on Finish

OBIEE 11g Architecture

OBIEE 11g Architecture

At this high level, components in OBIEE 11g can be divided into two types:
  • Java Components, the parts of OBIEE that are written in J2EE and run directly in the application server. These components include BI Publisher, the BI Office plugin, the BI Security Service, BI SOA Services, BI Action Services and the JMX MBeans that are used for configuring OBIEE
  • System Components, which were implemented as Windows services in 10g, and include the BI Server, BI Presentation Server, BI Scheduler and BI Cluster Controller





Web Logic Server


This part of OBIEE system contains Admin Server and Managed Server. Admin server is responsible for managing the start and stop processes for Managed server. Managed Server comprises of BI Plugin, Security, Publisher, SOA, BI Office, etc.

Node Manager

Node Manager triggers the auto start, stop, restart activities and provides process management activities for Admin and Managed server.

Oracle Process Manager and Notification Server (OPMN)

OPMN is used to start and stop all components of BI system. It is managed and controlled by Fusion Middleware Controller.

Oracle BI System Components

These are non-Java components in an OBIEE system.

Oracle BI Server

This is the heart of Oracle BI system and is responsible for providing data and query access capabilities.

BI Presentation Server

It is responsible to present data from BI server to web clients which is requested by the end users.

Scheduler

This component provides scheduling capability in BI system and it has its own scheduler to schedule jobs in OBIEE system.

Oracle BI Java Host

This is responsible for enabling BI Presentation server to support various Java tasks for BI Scheduler, Publisher and graphs.

BI Cluster Controller

This is used for load balancing purposes to ensure that the load is evenly assigned to all BI server processes.


Tuesday, April 11, 2017

File Locations in OBIEE 11G

File Locations in OBIEE 11G


Start BI Services (parameter:  start_all)
..\\\instances\instance1\bifoundation\OracleBIApplication\coreapplication\StartStopServices.cmd start_all

Stop BI Services (parameter:  stop_all)
..\\\instances\instance1\bifoundation\OracleBIApplication\coreapplication\StartStopServices.cmd stop_all

Sample Application Data and Repository
..\\\instances\instance1\bifoundation\OracleBIServerComponent\coreapplication_obis1\sample

BI Repository(RPD)
..\\\instances\instance1\bifoundation\OracleBIServerComponent\coreapplication_obis1\repository

Usage tracking and Marketing Segmentation Scripts
..\\\instances\instance1\bifoundation\OracleBIServerComponent\coreapplication_obis1\schema

BI Server Cache file
..\\\instances\instance1\bifoundation\OracleBIServerComponent\coreapplication_obis1\cache

BI Presentation Catalog
..\\\instances\instance1\bifoundation\OracleBIPresentationServicesComponent\coreapplication_obips1\catalog

Clustered Component
..\\\instances\instance1\bifoundation\OracleBIClusterControllerComponent\coreapplication_obiccs1

Scheduler Components (Scheduler schema etc.)
..\\\instances\instance1\bifoundation\OracleBISchedulerComponent\coreapplication_obisch1

BI ODBC , Javahost and other components
..\\\instances\instance1\bifoundation

Start Weblogic Server (startWebLogic.cmd)
..\\\user_projects\domains\bifoundation_domain

BI Config Files
..\\\instances\instance1\config

Repository Config files (NQSConfig.INI,DBFeatures.INI etc)
..\\\instances\instance1\config\OracleBIServerComponent\coreapplication_obis1

Catalog config files(instanceconfig.xml,credentialstore.xml etc)
..\\\instances\instance1\config\OracleBIPresentationServicesComponent\coreapplication_obips1

Scheduler Catalog config files (instanceconfig.xml)
..\\\instances\instance1\config\OracleBISchedulerComponent\coreapplication_obisch1

Cluster Config files
..\\\instances\instance1\config\OracleBIApplication\coreapplication

OPMN Startup (parameter:  startall)
..\\\instances\instance1\bin opmnctl startall

OPMN Stop (parameter:  stopall)
..\\\instances\instance1\bin opmnctl stopall

BI PATH Initialisation
..\\\instances\instance1\bifoundation\OracleBIApplication\coreapplication\setup\bi-init.cmd

Weblogic Admin and Managed Server start/stop scripts(startWebLogic.cmd,startManagedWebLogic.cmd etc.)
..\\\user_projects\domains\bifoundation_domain\bin

BIPublisher repository and Config files
..\\\user_projects\domains\bifoundation_domain\config\bipublisher

FMW Security Config files (system-jazn-data.xml)
..\\\user_projects\domains\bifoundation_domain\config\fmwconfig

BI Components Log files
..\\\instances\instance1\diagnostics\logs

BI Server Log files (NQserver.log , NQQuery.log etc.)
..\\\instances\instance1\diagnostics\logs\OracleBIServerComponent\coreapplication_obis1

BI Presentation Server Log files (sawlog)
..\\\instances\instance1\diagnostics\logs\OracleBIPresentationServicesComponent\coreapplication_obips1

Installation Log files
..\\\logs

Date Calculation Formulas OBIEE

Date Calculation Formulas

In OBIEE, we cannot perform an arithmetic operation on a DATE or TIMESTAMP. These are considered Non-numeric types.
To calculate difference between two dates, OBIEE provides a TIMESTAMPDIFF function. The syntax for using the function is:
TIMESTAMPDIFF(interval, first_date, second_date)
First_date and second_date have to be valid values in the TIMESTAMP type. Interval is a specified value.  Valid values for INTERVAL are:
  • SQL_TSI_SECOND
  • SQL_TSI_MINUTE
  • SQL_TSI_HOUR
  • SQL_TSI_DAY
  • SQL_TSI_WEEK
  • SQL_TSI_MONTH
  • SQL_TSI_QUARTER
  • SQL_TSI_YEAR
If the dates are not in a valid TIMESTAMP format, use CAST function to convert values into TIMESTAMP type. See example below:
TIMESTAMPDIFF(SQL_TSI_DAY, CAST(VALUEOF("MY_DATE_COLUMN") as TIMESTAMP), CURRENT_DATE)

1)    First Day of the Previous Year
-------------------------------------------
TIMESTAMPADD( SQL_TSI_YEAR , -1, TIMESTAMPADD( SQL_TSI_DAY , EXTRACT( DAY_OF_YEAR FROM CURRENT_DATE) * -(1) + 1, CURRENT_DATE)) 


From right to left the first TIMESTAMPADD returns the first day of the current year. The second TIMESTAMPADD removes a year from the returned date for the First Day of the Previous Year

2) First Day of the Current Year
----------------------------------
TIMESTAMPADD( SQL_TSI_DAY , EXTRACT( DAY_OF_YEAR FROM CURRENT_DATE) * -(1) + 1, CURRENT_DATE) 


This calculation returns the first day of the year by deducting one less than the total number of days in the year.

3) First Day of the Next Year
---------------------------------------
TIMESTAMPADD( SQL_TSI_YEAR , 1, TIMESTAMPADD( SQL_TSI_DAY , EXTRACT( DAY_OF_YEAR FROM CURRENT_DATE) * -(1) + 1, CURRENT_DATE))
From right to left the first TIMESTAMPADD returns the first day of the current year. The second TIMESTAMPADD adds a year to the date returned which will give the first day of the next year.

4) First Day of the Previous Month
-----------------------------------
TIMESTAMPADD(SQL_TSI_MONTH, -1, TIMESTAMPADD( SQL_TSI_DAY , DAYOFMONTH( CURRENT_DATE) * -(1) + 1, CURRENT_DATE))


 From right to left the first TIMESTAMPADD returns the first day of the Current Month. The second TIMESTAMPADD then subtracts one month from the first day of the Current Month arriving to the First Day of the previous month.

5) First Day of the Current Month
--------------------------------------------
TIMESTAMPADD( SQL_TSI_DAY , DAYOFMONTH( CURRENT_DATE) * -(1) + 1, CURRENT_DATE) This expression gets the current day of the month and subtracts one less than the current day to arrive at the first day of the month.

6) First Day of the Next Month
-------------------------------------
TIMESTAMPADD(SQL_TSI_MONTH, 1, TIMESTAMPADD( SQL_TSI_DAY , DAYOFMONTH( CURRENT_DATE) * -(1) + 1, CURRENT_DATE)) 


From right to left the first TIMESTAMPADD returns the first day of the Current Month. The second TIMESTAMPADD then adds one month from the first day of the Current Month arriving to the First Day of the next month.

7) First Day of Current Quarter
---------------------------------------
TIMESTAMPADD( SQL_TSI_DAY , DAY_OF_QUARTER( CURRENT_DATE) * -(1) + 1, CURRENT_DATE) 


This was included to show the calculations discussed above can be used with other functions. This is the same expression as the one that returns the first day of the current month except this one uses the DAY_OF_QUARTER property to return the first day of the current quarter.

8) Last Day of the Previous Month
--------------------------------------------------
TIMESTAMPADD( SQL_TSI_DAY , -(1), TIMESTAMPADD( SQL_TSI_DAY , DAYOFMONTH( CURRENT_DATE) * -(1) + 1, CURRENT_DATE)) 


From right to left the first TIMESTAMPADD returns the first day of the Current Month. The second TIMESTAMPADD subtracts a month to arrive at the first day of the previous month.

9) Last Day of Current Month
---------------------------------------
TIMESTAMPADD( SQL_TSI_DAY , -(1), TIMESTAMPADD( SQL_TSI_MONTH , 1, TIMESTAMPADD( SQL_TSI_DAY , DAYOFMONTH( CURRENT_DATE) * -(1) + 1, CURRENT_DATE))) 


From right to left the first TIMESTAMPADD finds the first day of the current Month. The second TIMESTAMPADD adds one month to the date to arrive at the first day of the next month. The final TIMESTAMPADD subtracts one day from the returned date to arrive at the last day of the Current Month.

10) Last Day of the Next Month
-----------------------------------------
TIMESTAMPADD( SQL_TSI_DAY , -(1), TIMESTAMPADD( SQL_TSI_MONTH , 2, TIMESTAMPADD( SQL_TSI_DAY , DAYOFMONTH( CURRENT_DATE) * -(1) + 1, CURRENT_DATE))) 


From right to left the first TIMESTAMPADD finds the first day of the current Month. The second TIMESTAMPADD adds two months to the date to arrive at the first day of month after next. The final TIMESTAMPADD subtracts one day from the returned date to arrive at the last day of the Next Month.

11) Last Day of Previous Year
------------------------------------------
TIMESTAMPADD( SQL_TSI_DAY , -1, TIMESTAMPADD( SQL_TSI_DAY , EXTRACT( DAY_OF_YEAR FROM CURRENT_DATE) * -(1) + 1,
CURRENT_DATE)) 


From right to left the first TIMESTAMPADD returns the first day of the current year. The second TIMESTAMPADD subtracts one day to arrive at December 31st of the previous year.

12) Last Day of Current Year
------------------------------------------
TIMESTAMPADD(SQL_TSI_YEAR, 1, TIMESTAMPADD( SQL_TSI_DAY , -1, TIMESTAMPADD( SQL_TSI_DAY , EXTRACT( DAY_OF_YEAR FROM CURRENT_DATE) * -(1) + 1, CURRENT_DATE))) 


From right to left the first TIMESTAMPADD returns the first day of the current year. The second TIMESTAMPADD deducts one day to arrive at December 31 of the previous year. The third TIMESTAMPADD adds a single year to the date to arrive at December 31 of the Current Year.

13) Last Day of the Next Year
----------------------------------------
TIMESTAMPADD(SQL_TSI_YEAR, 2, TIMESTAMPADD( SQL_TSI_DAY , -1, TIMESTAMPADD( SQL_TSI_DAY , EXTRACT( DAY_OF_YEAR FROM CURRENT_DATE) * -(1) + 1, CURRENT_DATE))) 


From right to left the first TIMESTAMPADD returns the first day of the current year. The second TIMESTAMPADD deducts one day to arrive at December 31 of the previous year. The third TIMESTAMPADD adds 2 years to the date to arrive at December 31 of the Next Year.

14) Last Day of Current Quarter
--------------------------------------------
TIMESTAMPADD( SQL_TSI_DAY , -(1), TIMESTAMPADD( SQL_TSI_QUARTER , 1, TIMESTAMPADD( SQL_TSI_DAY , DAY_OF_QUARTER( CURRENT_DATE) * -(1) + 1, CURRENT_DATE))) 


Demonstrated using Quarters. From right to left the first TIMESTAMPADD returns the first day of the Current Quarter. The second TIMESTAMPADD returns the first day of the next quarter. The final TIMESTAMPADD subtracts a single day from the date to arrive at the last day of the Current Quarter.

15) Number of days between First Day of Year and Last Day of Current Month
------------------------------------------------------------------------------------------------------------------
 TIMESTAMPDIFF(SQL_TSI_DAY, CAST('2010/01/01 00:00:00' AS DATE), TIMESTAMPADD( SQL_TSI_DAY , -(1), TIMESTAMPADD( SQL_TSI_MONTH , 1, TIMESTAMPADD( SQL_TSI_DAY , DAYOFMONTH( CURRENT_DATE) * -(1) + 1, CURRENT_DATE)))) 


For simplicity I hard coded the January 1, 2010 date and CAST it to a date. I could have used the First Day of the Current Year calculation but didn’t want to over clutter the example. The second part of the TIMESTAMPDIFF uses Last Day of the Current Month calculation to force the TIMESTAMPDIFF to calculate the number of days between the first day of the year and the last day of the current month.
16) Lat Date of Week:

TIMESTAMPADD(SQL_TSI_DAY,DAYOFWEEK(Current_Date)*-1,TIMESTAMPADD(SQL_TSI_WEEK, 1, Current_date))


17) First Date of Year:

TIMESTAMPADD( SQL_TSI_DAY , EXTRACT( DAY_OF_YEAR FROM CURRENT_DATE) * -(1) + 1, CURRENT_DATE)

18) Last Date of Year:

TIMESTAMPADD(SQL_TSI_YEAR, 1, TIMESTAMPADD( SQL_TSI_DAY , -1, TIMESTAMPADD( SQL_TSI_DAY , EXTRACT( DAY_OF_YEAR FROM CURRENT_DATE) * -(1) + 1, CURRENT_DATE)))

How to Manage OBIEE services Manually

How to Manage OBIEE services Manually



Start your Oracle Database

Start Weblogic Node Manager
ORACLE_BIEE_HOME/wlserver_10.3/server/bin/startNodeManager.cmd

Steps to Manually Start Oracle BIEE 11g

1. ORACLE_BIEE_HOME/user_projects/domains/bifoundation_domain/bin/startWebLogic.cmd

2. ORACLE_BIEE_HOME/user_projects/domains/bifoundation_domain/bin/startManagedWebLogic.cmd bi_server1

3. ORACLE_BIEE_HOME/instances/instance1/bin/opmnctl startall

Steps to Manually Stop Oracle BIEE 11g

1. ORACLE_BIEE_HOME/instances/instance1/bin/opmnctl stopall

2. ORACLE_BIEE_HOME/user_projects/domains/bifoundation_domain/bin/stopManagedWebLogic.cmd bi_server1

3.ORACLE_BIEE_HOME/user_projects/domains/bifoundation_domain/bin/stopWebLogic.cmd

Monday, March 20, 2017

Informatica PowerCenter 9 Architecture


Informatica PowerCenter 9 Architecture




Before we learn how to use Informatica, we need to understand what are the important components of Informatica and how it works.
Informatica tool consists of following services & components
1.Repository Service – Responsible for maintaining Informatica metadata & providing access of same to other services.
2.Integration Service – Responsible for the movement of data from sources to targets
3.Reporting Service - Enables the generation of reports
4.Nodes – Computing platform where the above services are executed
5.      Informatica Designer - Used for creation of mappings between source and target
6.      Workflow Manager – Used to create workflows and other task & their execution
7.      Workflow Monitor – Used to monitor the execution of workflows

8.      Repository Manager – Used to manage objects in repository


Informatica Repository:
The informatica repository is at the center of the informatica suite.
·        The Informatica repository is a relational database that stores information, or metadata, used by the Informatica Server and Client tools.
·        Metadata is data about data which include information such as source definitions , target definitions, mappings describing how to transform source data, sessions indicating when you want the Informatica Server to perform the transformations, and connect strings for sources and targets. 
·        The repository also stores administrative information such as usernames and passwords, permissions and privileges, and product version. 
·        Use repository manager to create the repository. The Repository Manager connects to the repository database and runs the code needed to create the repository tables. These tables stores metadata in specific format the informatica server, client tools use. 



Informatica Repository:
The informatica repository is at the center of the informatica suite.
·        The Informatica repository is a relational database that stores information, or metadata, used by the Informatica Server and Client tools.
·        Metadata is data about data which include information such as source definitions , target definitions, mappings describing how to transform source data, sessions indicating when you want the Informatica Server to perform the transformations, and connect strings for sources and targets. 
·        The repository also stores administrative information such as usernames and passwords, permissions and privileges, and product version. 
·        Use repository manager to create the repository. The Repository Manager connects to the repository database and runs the code needed to create the repository tables. These tables stores metadata in specific format the informatica server, client tools use. 
 Informatica Components: 
Server Components:
  1. Repository Service
  2. Integration Service

Client Components:
  1. Repository Manager
  2. Designer
  3. Workflow Manager
  4. Workflow Monitor

Server Components 
1. Repository Server:
Ø      The Repository Server manages the metadata in the repository database.
Ø      The Repository Server manages connections to the repository from client applications.
Ø      The Repository Service is a separate, multi-threaded process that retrieves, inserts, and updates metadata in the repository database tables. The Repository Service ensures the consistency of metadata in the repository.

2. Integration Service:
Ø      The Integration Server reads mapping and session information from the repository. It extracts data from the mapping sources and stores the data in memory while it applies the transformation rules that you configure in the mapping. The Integration Server loads the transformed data into the mapping targets.
Ø      Manages the scheduling and execution of workflows
Ø      The Integration Server can start and run multiple workflows concurrently. It can also concurrently process partitions within a single session. 


Monday, March 13, 2017

Seeding the Cache IN OBIEE11G

Seeding the Cache

I runs an OBIEE11G dashboard, and the results are added to the cache so that when Bill runs the same dashboard Bill gets a great response rate because his dashboard runs straight from cache. Kinda sucks for Bob though, because his query ran slow as it wasn’t in the cache yet. What’d be nice would be that for the first user on a dashboard the results were already in cache. There are several options for seeding the cache. These all assume you’ve figured out the queries that you want to run in order to load the results into cache.
  1. Run the analysis manually, which will return the analysis data to you and insert it into the BI Server Cache too.
  2. Create an Agent to run the analysis with destination set to Oracle BI Server Cache (For seeding cache), and then either:
    1. Schedule the analysis to run from an Agent on a schedule
    2. Trigger it from a Web Service in order to couple it to your ETL data load / cache purge batch steps.
  3. Use the BI Server Procedure SASeedQuery (which is what the Agent does in the background) to load the given query into cache without returning the data to the client. This is useful for doing over JDBC/ODBC/Web Service (as discussed for purging above). You could just run the Logical SQL itself, but you probably don’t want to pull the actual data back to the client, hence using the procedure call instead.
  4. SET VARIABLE SAW_SRC_PATH='/users/weblogic/Cache Test 01',  
    DISABLE\_CACHE\_HIT=1:SELECT  
       0 s_0,  
       "A - Sample Sales"."Time"."T02 Per Name Month" s_1,  
       "A - Sample Sales"."Base Facts"."1- Revenue" s_2  
    FROM "A - Sample Sales"  
    ORDER BY 1, 2 ASC NULLS LAST  
    FETCH FIRST 5000001 ROWS ONLY  

Checking the RPD for Cacheable Tables

The RPD Query Tool is great for finding objects matching certain criteria. However, it seems to invert results when looking for Cacheable Physical tables - if you add a filter of Cacheable = false you get physical tables where Cacheable is enabled! And the same in reverse (Cacheable = true -> shows Physical tables where Cacheable is disabled)

Cache Location

The BI Server cache is held on disk, so it goes without saying that storing it on fast (eg SSD) disk is a Good Idea. There's no harm in giving it its own filesystem on *nix to isolate it from other work (in terms of filesystems filling up) and to make monitoring it super easy.
Use the DATASTORAGEPATHS configuration element in NQSConfig.ini to change the location of the BI Server cache

Data engineering Interview Questions

1)  What all challenges you have faced and how did you overcome from it? Ans:- Challenges Faced and Overcome As a hypothetical Spark develop...