{"id":1632,"date":"2026-03-09T19:51:40","date_gmt":"2026-03-09T19:51:40","guid":{"rendered":"https:\/\/adrianotanaka.com.br\/?p=1632"},"modified":"2026-03-09T20:01:28","modified_gmt":"2026-03-09T20:01:28","slug":"architecting-the-real-time-lakehouse-from-goldengate-ingestion-to-oracle-aidp","status":"publish","type":"post","link":"https:\/\/adrianotanaka.com.br\/index.php\/2026\/03\/09\/architecting-the-real-time-lakehouse-from-goldengate-ingestion-to-oracle-aidp\/","title":{"rendered":"Architecting the Real-Time Lakehouse: From\u00a0GoldenGate\u00a0Ingestion to Oracle AIDP"},"content":{"rendered":"\n

Architecting the Real-Time Lakehouse: From GoldenGate Ingestion to Oracle AIDP<\/strong><\/p>\n\n\n\n

Oracle AI Data Platform or simply AIDP is a platform where companies could build their Lakehouse\u2019s using bleeding-edge technologies like AI Agents, Open Table formats and Spark as the engine of their processing layer. <\/p>\n\n\n\n

Adding Oracle GoldenGate to this topology is the right way to bring near real time data movement capabilities, in this article I will talk about both technologies, how Oracle GoldenGate interacts with AIDP, what are the requirements and so on.<\/p>\n\n\n\n

AIDP Architecture<\/strong><\/h2>\n\n\n\n

Before diving into the replication process, it\u2019s important to understand the key components of AIDP architecture:<\/p>\n\n\n\n

Catalog<\/strong><\/h5>\n\n\n\n

The AIDP Catalog is a resource that centralizes the metadata of your datasets, doing a parallel within Oracle Database, like the Data Dictionary, it allows you to setup the data governance and other features like mapping and managing structured and unstructured data (like files in Object Storage)<\/p>\n\n\n\n

Schema<\/strong><\/h5>\n\n\n\n

In the schema there will be the logical structure of your datasets just like a schema from Oracle Database, you can have tables and views.<\/p>\n\n\n\n

Table<\/strong><\/h5>\n\n\n\n

Here we can see one of the most important AIDP features, the support for Delta table format, using Delta table we can have ACID features with \u201cflat files\u201d in a cloud storage<\/p>\n\n\n\n

Compute<\/strong><\/h5>\n\n\n\n

The Compute resource from AIDP is responsible for delivering Spark Environment, you can use CPU org GPU machines, divided in Drivers and Workers nodes (you can even have multiple workers if your workload needs to).<\/p>\n\n\n\n

AIDP has more components but for GoldenGate we will use the previous ones:<\/p>\n\n\n\n

The replicat process will connect to a JDBC endpoint (it runs inside the compute cluster), it will use the three-part namespace (Catalog.Schema.Table) to identify the target object and execute a stage-merge procedure.<\/p>\n\n\n\n

Creating the required resources in AIDP:<\/strong><\/h5>\n\n\n\n

First, we will create a Catalog :<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

It may be a Standard Catalog type.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

Now let’s create our schema, here I will be using the medallion architecture (Bronze, Silver and Gold) and my schema will be \u201cbronze\u201d:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The workspace is used to organize your resources, mainly the python notebooks but could be used to build workflows and most importantly, inside your workspace you will have the compute cluster that provides the compute power to run your workloads.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

When deploying your Compute cluster you can define the size of it and the python version, for whom is used to work with Spark it\u2019s easy to spot the similarity, here you have the Drivers and Workers nodes.<\/p>\n\n\n\n

After the cluster creation, go to Connection Details tab, there you will see the JDBC URL, we need it to setup the GoldenGate connection.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
Data Delivery<\/strong><\/h5>\n\n\n\n

Oracle GoldenGate has multiple formats when we are talking about data delivery, here we will work with Stage and Merge replicat, where our data is stored inside a OCI Object Storage and merged into Delta tables inside AIDP, but you can send data to a bucket in avro\/parquet file and ingest into AIDP table using PySpark.<\/p>\n\n\n\n

Connections<\/strong><\/h5>\n\n\n\n

I\u2019m using the OCI GoldenGate but for the on-premises version it should be similar, you need to use the JDBC endpoint with tenancy and user OCID, region, just like a normal OCI authentication, you can always refer to official documents https:\/\/docs.oracle.com\/en\/middleware\/goldengate\/big-data\/23\/gadbd\/qs-realtime-data-ingestion-oracle-ai-data-platform-oracle-goldengate-daa.html#GUID-52A45276-7087-432E-ADC3-76A48D2E4A9E<\/a> if you have doubts.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Besides the AIDP connection you will need an OCI Object Storage connection.<\/p>\n\n\n\n

Extract<\/strong><\/h2>\n\n\n\n

For the extract I suggest you use these parameters:<\/p>\n\n\n\n

ddl include mapped - -> To capture DDL from mapped objects<\/code><\/pre>\n\n\n\n
LOGALLSUPCOLS - -> To capture all columns, we will need this in the target side<\/code><\/pre>\n\n\n\n
Replicat<\/strong><\/h2>\n\n\n\n
Stage and Merge<\/strong><\/h5>\n\n\n\n
Using the default replicat, where we write directly into AIDP tables:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
In your properties file define the temporary bucket and the compartment with these parameters:<\/p>\n\n\n\n
gg.eventhandler.oci.compartmentID=ocid1.compartment.oc1..aaaXXXX<\/code><\/pre>\n\n\n\n
gg.eventhandler.oci.bucketMappingTemplate=bucket-tmp<\/code><\/pre>\n\n\n\n
In parameter file we must configure the Catalog.Schema mapping, something like this:<\/p>\n\n\n\n
MAP *.*, TARGET catalog_tanaka.bronze.*;<\/code><\/pre>\n\n\n\n
In this case the temporary dat file will be generated in bucket-tmp bucket and the final data will be written into the catalog_tanaka catalog inside the bronze schema, using the specified JDBC endpoint from connection.<\/p>\n\n\n\n
If everything is correctly configured when can track the steps from Spark UI (Cluster > Spark UI option > SQL \/ DataFrame tab):<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
As you can see, first the replicat will see if the table exists (SHOW tables command) and starts to generate the temporary files into the bucket:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
The table will be created, note the USING DELTA parameter:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
And finally, the data will be merged:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
And we can use a Python Notebook to query it:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Because I\u2019m using a wildcard in the map parameter, I just need to trigger a DML from my source DB and It will replicate the table:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Did you notice that _stage_ tables? They\u2019re here to do the merge from the Object Storage files.<\/p>\n\n\n\n
DDL Replication<\/strong><\/h5>\n\n\n\n
As of today, there is no DDL replication available for AIDP targets, so you must choose what action to take when a DDL happens at the source.<\/p>\n\n\n\n
Use the DDLOPTIONS REPORT parameter in your replicat parameter to print the DDL and you can work with EVENTACTIONS to ignore (STOP) or abend the process(ABORT).<\/p>\n\n\n\n
With this combination of parameters:<\/p>\n\n\n\n
DDL INCLUDE ALL EVENTACTIONS (REPORT, ABORT)  EXCLUDE OPTYPE CREATE<\/code><\/pre>\n\n\n\n
DDLOPTIONS REPORT<\/code><\/pre>\n\n\n\n
You will have this in your discard file:<\/p>\n\n\n\n
Operation discarded due to ABORT event from DDL on object name tanaka.MINHA_TAB_PERF3 in file \/u02\/Deployment\/var\/lib\/data\/aidp\/aa000000001, RBA 6789660<\/code><\/pre>\n\n\n\n
SQL Operation [alter table TNK_SOURCE.MINHA_TAB_PERF3 add (cpf2 varchar2(10)) (size 62)]<\/code><\/pre>\n\n\n\n
So, it should be easy to fix that issue.<\/p>\n\n\n\n
<\/p>\n\n\n\n
The integration allows organizations to leverage GoldenGate’s proven replication power to feed a high-performance Spark\/LakeHouse environment, enabling real-time analytics and AI-ready data layers within the Oracle ecosystem.<\/p>\n","protected":false},"excerpt":{"rendered":"
Architecting the Real-Time Lakehouse: From GoldenGate Ingestion to Oracle AIDP Oracle AI Data Platform or simply AIDP is a platform where companies could build their Lakehouse\u2019s using bleeding-edge technologies like AI Agents, Open Table formats and Spark as the engine of their processing layer.  Adding Oracle GoldenGate to this topology is the right way to bring near […]<\/p>\n","protected":false},"author":1,"featured_media":1661,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"material-hide-sections":[],"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1632","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/adrianotanaka.com.br\/index.php\/wp-json\/wp\/v2\/posts\/1632","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/adrianotanaka.com.br\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/adrianotanaka.com.br\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/adrianotanaka.com.br\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/adrianotanaka.com.br\/index.php\/wp-json\/wp\/v2\/comments?post=1632"}],"version-history":[{"count":11,"href":"https:\/\/adrianotanaka.com.br\/index.php\/wp-json\/wp\/v2\/posts\/1632\/revisions"}],"predecessor-version":[{"id":1662,"href":"https:\/\/adrianotanaka.com.br\/index.php\/wp-json\/wp\/v2\/posts\/1632\/revisions\/1662"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/adrianotanaka.com.br\/index.php\/wp-json\/wp\/v2\/media\/1661"}],"wp:attachment":[{"href":"https:\/\/adrianotanaka.com.br\/index.php\/wp-json\/wp\/v2\/media?parent=1632"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/adrianotanaka.com.br\/index.php\/wp-json\/wp\/v2\/categories?post=1632"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/adrianotanaka.com.br\/index.php\/wp-json\/wp\/v2\/tags?post=1632"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}