Change data capture (CDC) refers to the process of identifying and capturing changes made to data in a database and then delivering those changes in real-time to a downstream process or system. Capturing every change from transactions in a source database and moving them to the target in real-time keeps…

In Oracle, cursors plays major role in PL/SQL blocks or Procedures to iterate through the table or datasets, there was always a question how the same can be implemented in BigQuery SQL specially while writing procedural language code, so this can be achieved by using for — do — end…

This article is specifically to explain how failure notifications can be sent to MS teams channel from cloud composer. In Cloud Composer (Airflow) we run DAGs, you can check my below blog on Cloud Composer to understand more on DAG’s. Google Cloud Composer (built on Apache Airflow) Simplified Google Cloud Platform gives a service called Cloud Composer to schedule, and monitor pipelines that span across hybrid…nileshk611.medium.com

Cross DAG dependency or sensing the completion of external airflow dags or tasks can be very useful when you have dependency between DAGs or task in a different DAG to complete for a specific execution date/time. This can be achieved using airflow operator called as ExternalTaskSensor. Sensors are pre-built in…

Firstly lets understand what is Data Catalog, it is a fully managed, scalable metadata management service offering from GCP. Data Catalog can be used for searching insightful data like what data is where, understanding data like what is the source or who is the owner and also understanding data which…

Database is either Column oriented database- or row oriented database, choosing either of this will have an impact on your application performance and will solely depend on your use-case.

Google Cloud DLP provides tools to classify, mask, tokenize, and transform sensitive elements to help you better manage the data that you collect, store, or use for business or analytics. DLP mainly Inspects i.e. find sensitive data and De-identify i.e. masks the sensitive data. This article will mainly focus to…

This article talks about various transfer option offerings by Google Cloud, there are various options given by Google Cloud to efficiently transfer the data. The data transfer can be from on-prem data to cloud or between cloud to cloud i.e. GCP to GCP or any other cloud to GCP. Below…

In the database world, we have heard about simple views, complex views and even materialized views, however BigQuery has come up with something called as Authorized Views. A lot of organizations turn to BigQuery as a datawarehouse solution in the cloud. But as the amount of data increases and the…